Tales From A Lazy Fat DBA

Den of an Oracle DB Performance freak & a fan of Cassandra, MySQL and PostgreSQL …

  • Likes

    • 138,422
  • Archives

  • Categories

  • Subscribe

  • Advertisements
  • Prashant Dixit is the FatDBA

  • Follow Tales From A Lazy Fat DBA on WordPress.com
  • My Twitter Feeds

  • Disclaimer!

    FatDBA or Oracle ‘Ant’ is an independent web-blog/site.The experiences, Test cases, views, and opinions expressed in this website are my own and does not reflect the views or opinions of my employer.

    This site is independent of and does not represent Oracle Corporation in any way. Oracle does not officially sponsor, approve, or endorse this site or its content.
    Product and company names mentioned in this website may be the trademarks of their respective owners.

Posts Tagged ‘exadata’

Collecting Exadata Cell Performance Statistics using ‘Cellperfdiag.sql’

Posted by FatDBA on July 22, 2018

Hi Guys,

Last week we finished migration for one of the customer who recently purchased the Oracle Exadata Machines (X6-2), after that complex data movement is successfully completed we observed some great performance improvements without actually making any changes and exadata features in action.

But its not that the Exa machines are ‘In-frangible’ Or Unbreakable. There are many of the times when you actually have to understand the Cell/Computer Nodes or system wide performance and output from tools like Sundiag, Exawatcher, Exachk, TFA, SOS report etc. were not sufficient.

For the purpose of ‘Cell Performance DIAG info’ Oracle has provided a script to collect Exadata Cell Performance statistics which you can use to interpret the issues.

Output of the script is distributed into several sections
1. Cell specific parameters
2. Top CELL waits with wait times (Worst Times)
3. Cell Configuration details

Script to Collect Exadata Cell Performance Information (cellperfdiag.sql) (Doc ID 2038493.1)
Below is the output from one of the Cell Node of a lower environment.


CELLPERFDIAG DATA FOR DixitLab_July19_0257

IMPORTANT PARAMETERS RELATING TO CELL PERFORMANCE:

INST_ID NAME                                     VALUE
------- ---------------------------------------- ----------------------------------------
      1 cell_offload_compaction                  ADAPTIVE
      1 cell_offload_decryption                  TRUE
      1 cell_offload_plan_display                PDTO
      1 cell_offload_processing                  TRUE
      2 cell_offload_compaction                  ADAPTIVE
      2 cell_offload_decryption                  TRUE
      2 cell_offload_plan_display                PDTO
      2 cell_offload_processing                  TRUE

TOP 20 CURRENT CELL WAITS

This is to look at current cell waits, July not return any data.

ASH CELL PERFORMANCE SUMMARY

This query will look at the average cell wait times for each cell in ASH

CELL_PATH                      TOTAL_WAIT_TIME  AVG_WAIT_TIME
------------------------------ --------------- --------------
192.199.66.19;192.199.66.20       1185605081.0        96124.9
192.199.66.21;192.199.66.22       1148823479.0        88371.0
192.199.66.17;192.199.66.18       1048776677.0        82115.3
192.199.66.23;192.199.66.24        927836650.0        76604.7

20 WORST CELL PERFORMANCE MINUTES IN ASH:

APPROACH: These are the minutes where the avg cell perf time
was the highest.  See which cell had the longest waits and
during what minute.

MINUTE       CELL_PATH                      TOTAL_WAIT_TIME  AVG_WAIT_TIME
------------ ------------------------------ --------------- --------------
July18_1308   192.199.66.21;192.199.66.22         28445866.0       536714.5
July18_1158   192.199.66.19;192.199.66.20         25685919.0       524202.4
July18_1218   192.199.66.23;192.199.66.24           496960.0       496960.0
July18_2008   192.199.66.21;192.199.66.22         21481465.0       488215.1
July18_1618   192.199.66.19;192.199.66.20         56796413.0       485439.4
July18_1638   192.199.66.17;192.199.66.18         23823342.0       467124.4
July18_2004   192.199.66.21;192.199.66.22         28935643.0       466703.9
July18_1207   192.199.66.17;192.199.66.18          1779234.0       444808.5
July19_0042   192.199.66.19;192.199.66.20           888334.0       444167.0
July18_1654   192.199.66.23;192.199.66.24          2619836.0       436639.3
July18_2008   192.199.66.19;192.199.66.20         11634865.0       430920.9
July18_1324   192.199.66.19;192.199.66.20         66537869.0       423808.1
July18_1157   192.199.66.21;192.199.66.22         22725628.0       420845.0
July19_0028   192.199.66.21;192.199.66.22           841081.0       420540.5
July18_1323   192.199.66.19;192.199.66.20         76790471.0       419620.1
July18_1157   192.199.66.19;192.199.66.20         28103203.0       401474.3
July18_1159   192.199.66.17;192.199.66.18         14050389.0       401439.7
July18_1158   192.199.66.23;192.199.66.24         10054891.0       386726.6
July18_1639   192.199.66.17;192.199.66.18         79265611.0       386661.5
July18_2009   192.199.66.21;192.199.66.22         20335679.0       383692.1

50 LONGEST CELL WAITS IN ASH ORDERED BY WAIT TIME

APPROACH: These are the top 50 individual cell waits in ASH
in wait time order.

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
19/JULY/18 08:04:19.964 PM 192.199.66.21;192.199.66.22          2 cell smart incremental backup            2990700133          0          0     5560085
19/JULY/18 08:04:18.964 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 4112515153      32768     3333391
19/JULY/18 04:38:52.666 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2653079
19/JULY/18 04:38:51.656 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2486538
19/JULY/18 01:23:19.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2461036
19/JULY/18 04:38:51.656 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2447901
19/JULY/18 11:59:13.149 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2443358
19/JULY/18 11:57:14.811 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3718979034      32768     2433342
19/JULY/18 02:53:14.279 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440   24718082      32768     2431871
19/JULY/18 01:24:36.080 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2430882
19/JULY/18 01:36:57.174 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3409456992      32768     2429672
19/JULY/18 01:36:58.527 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3530077886      32768     2427505
19/JULY/18 01:36:56.527 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     2427165
19/JULY/18 11:59:13.149 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2426099
19/JULY/18 01:24:40.011 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131  931391314      32768     2417661
19/JULY/18 01:24:40.090 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2416163
19/JULY/18 01:24:47.100 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2396291
19/JULY/18 08:04:36.984 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     2388370
19/JULY/18 01:23:36.971 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2370848
19/JULY/18 04:48:26.040 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213  842883291      32768     2359311
19/JULY/18 03:35:36.076 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2367494631      32768     2344972
19/JULY/18 01:36:56.527 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2330945
19/JULY/18 11:57:52.881 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2310010
19/JULY/18 04:38:55.395 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2304803
19/JULY/18 01:37:01.537 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3530077886      32768     2284322
19/JULY/18 01:23:19.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2275197
19/JULY/18 01:36:58.527 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3530077886      32768     2250389
19/JULY/18 11:58:11.911 AM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2256514768      32768     2176213
19/JULY/18 01:23:46.981 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2149317
19/JULY/18 01:24:54.380 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 1783777120      32768     2138193
19/JULY/18 01:24:14.051 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     2107128
19/JULY/18 08:07:04.178 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1233700156      32768     2102692
19/JULY/18 08:04:28.984 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     2098081
19/JULY/18 02:43:04.647 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2367494631      32768     2090428
19/JULY/18 01:23:47.921 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2077418
19/JULY/18 01:24:48.100 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2056411
19/JULY/18 01:24:48.021 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131  931391314      32768     2043911
19/JULY/18 11:57:57.891 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2033684
19/JULY/18 08:03:38.870 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 2251483371      32768     2032547
19/JULY/18 01:37:01.174 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2031799
19/JULY/18 08:03:39.874 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  474454945      32768     2024898
19/JULY/18 11:58:31.931 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  474454945      32768     2013268
19/JULY/18 03:53:00.757 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2701596410      32768     2004020
19/JULY/18 04:39:03.405 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     1997819
19/JULY/18 04:18:40.331 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     1994866
19/JULY/18 01:23:34.861 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1467922874      32768     1988301
19/JULY/18 04:38:57.666 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440   24718082      32768     1985840
19/JULY/18 01:23:17.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     1983376
19/JULY/18 01:08:56.066 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3835758527      32768     1965134
19/JULY/18 04:39:39.763 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     1956491

100 LONGEST CELL WAITS IN ASH ORDERED BY SAMPLE TIME

APPROACH: These are the top 50 individual cell waits in ASH
in sample time order.

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
19/JULY/18 11:57:14.811 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3718979034      32768     2433342
19/JULY/18 11:57:27.821 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  552535141      32768     1627154
19/JULY/18 11:57:49.881 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     1871961
19/JULY/18 11:57:52.881 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     1634342
19/JULY/18 11:57:52.881 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2310010
19/JULY/18 11:57:57.891 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2033684
19/JULY/18 11:58:11.911 AM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2256514768      32768     2176213
19/JULY/18 11:58:31.931 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  474454945      32768     2013268
19/JULY/18 11:58:56.001 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1433462179      32768     1665222
19/JULY/18 11:59:11.149 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 3958490174      32768     1796201
19/JULY/18 11:59:13.149 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2426099
19/JULY/18 11:59:13.149 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2443358
19/JULY/18 12:02:25.487 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 1783777120      32768     1667222
19/JULY/18 12:36:22.441 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 2891233531      32768     1817473
19/JULY/18 01:08:11.966 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     1824501
19/JULY/18 01:08:53.056 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  758773684      32768     1633915
19/JULY/18 01:08:56.066 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3835758527      32768     1965134
19/JULY/18 01:22:18.849 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     1692084
19/JULY/18 01:22:35.760 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1404181515      32768     1927461
19/JULY/18 01:22:35.760 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1404181515      32768     1892900
19/JULY/18 01:22:35.871 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     1817924
19/JULY/18 01:23:08.830 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3835758527      32768     1798727
19/JULY/18 01:23:17.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     1983376
19/JULY/18 01:23:19.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2461036
19/JULY/18 01:23:19.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2275197
19/JULY/18 01:23:19.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1340519488      32768     1600315
19/JULY/18 01:23:21.951 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     1806676
19/JULY/18 01:23:32.971 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     1916487
19/JULY/18 01:23:34.861 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1467922874      32768     1988301
19/JULY/18 01:23:36.971 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2370848
19/JULY/18 01:23:46.981 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2149317
19/JULY/18 01:23:47.921 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2077418
19/JULY/18 01:24:09.051 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     1716730
19/JULY/18 01:24:14.051 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     2107128
19/JULY/18 01:24:36.080 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2430882
19/JULY/18 01:24:40.011 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131  931391314      32768     2417661
19/JULY/18 01:24:40.090 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2416163
19/JULY/18 01:24:47.100 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2396291
19/JULY/18 01:24:48.021 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131  931391314      32768     2043911
19/JULY/18 01:24:48.100 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2056411
19/JULY/18 01:24:54.380 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 1783777120      32768     2138193
19/JULY/18 01:35:40.244 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2381222600      32768     1670138
19/JULY/18 01:36:56.527 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2330945
19/JULY/18 01:36:56.527 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     2427165
19/JULY/18 01:36:57.174 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3409456992      32768     1872516
19/JULY/18 01:36:57.174 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3409456992      32768     2429672
19/JULY/18 01:36:58.527 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3530077886      32768     2250389
19/JULY/18 01:36:58.527 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3530077886      32768     2427505
19/JULY/18 01:37:01.174 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2031799
19/JULY/18 01:37:01.174 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     1782826
19/JULY/18 01:37:01.537 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3530077886      32768     2284322
19/JULY/18 01:37:04.587 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     1609551
19/JULY/18 02:43:01.717 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     1705580
19/JULY/18 02:43:04.647 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2367494631      32768     2090428
19/JULY/18 02:43:04.757 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     1667922
19/JULY/18 02:43:11.657 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  552535141      32768     1700677
19/JULY/18 02:53:14.279 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440   24718082      32768     2431871

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
19/JULY/18 03:09:17.091 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440   24718082      32768     1718584
19/JULY/18 03:09:22.101 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1404181515      32768     1924659
19/JULY/18 03:09:23.820 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     1609377
19/JULY/18 03:35:36.076 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2367494631      32768     2344972
19/JULY/18 03:42:53.196 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     1705247
19/JULY/18 03:43:09.256 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1340519488      32768     1621901
19/JULY/18 03:53:00.757 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2701596410      32768     2004020
19/JULY/18 03:53:03.356 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2701596410      32768     1951623
19/JULY/18 03:53:04.767 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2701596410      32768     1911016
19/JULY/18 04:02:42.355 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 1122005524      32768     1600296
19/JULY/18 04:18:40.331 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     1994866
19/JULY/18 04:18:41.331 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     1680223
19/JULY/18 04:18:47.341 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     1676801
19/JULY/18 04:38:51.656 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2447901
19/JULY/18 04:38:51.656 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2486538
19/JULY/18 04:38:52.666 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2653079
19/JULY/18 04:38:55.395 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2304803
19/JULY/18 04:38:57.666 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440   24718082      32768     1985840
19/JULY/18 04:39:03.405 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     1997819
19/JULY/18 04:39:26.743 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 1433462179      32768     1686065
19/JULY/18 04:39:39.455 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     1770436
19/JULY/18 04:39:39.763 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     1956491
19/JULY/18 04:39:40.763 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1122005524      32768     1719451
19/JULY/18 04:39:45.763 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131   91591739      32768     1669777
19/JULY/18 04:39:53.783 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     1606474
19/JULY/18 04:47:20.930 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 1467922874      32768     1598316
19/JULY/18 04:47:21.101 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1467922874      32768     1723089
19/JULY/18 04:48:26.040 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213  842883291      32768     2359311
19/JULY/18 05:17:36.317 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  125724152      32768     1743703
19/JULY/18 05:23:11.293 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  552535141      32768     1631948
19/JULY/18 05:23:19.727 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     1741381
19/JULY/18 06:23:57.158 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1374369774      32768     1769798
19/JULY/18 08:03:38.870 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2701596410      32768     1622627
19/JULY/18 08:03:38.870 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 2251483371      32768     2032547
19/JULY/18 08:03:39.874 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  474454945      32768     2024898
19/JULY/18 08:03:39.874 PM 192.199.66.23;192.199.66.24          2 cell smart incremental backup            3802949213          0          0     1775912
19/JULY/18 08:04:18.964 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 4112515153      32768     3333391
19/JULY/18 08:04:19.964 PM 192.199.66.21;192.199.66.22          2 cell smart incremental backup            2990700133          0          0     5560085
19/JULY/18 08:04:28.984 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     2098081
19/JULY/18 08:04:36.984 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     2388370
19/JULY/18 08:07:04.178 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1233700156      32768     2102692
19/JULY/18 08:09:11.705 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  870191132      32768     1753415
19/JULY/18 10:13:12.946 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     1777250

ASH HISTORY CELL PERFORMANCE SUMMARY

This query will look at the average cell wait times for each cell in ASH

CELL_PATH                      TOTAL_WAIT_TIME  AVG_WAIT_TIME
------------------------------ --------------- --------------
192.199.66.19;192.199.66.20       4320370317.0       144803.9
192.199.66.17;192.199.66.18       3720276327.0       126342.3
192.199.66.21;192.199.66.22       3302364789.0       116198.6
192.199.66.23;192.199.66.24       2214253857.0        86457.1

20 WORST CELL PERFORMANCE MINUTES IN ASH HISTORY:

APPROACH: These are the minutes where the avg cell perf time
was the highest.  See which cell had the longest waits and
during what time minute.

MINUTE       CELL_PATH                      TOTAL_WAIT_TIME  AVG_WAIT_TIME
------------ ------------------------------ --------------- --------------
July17_0458   192.199.66.21;192.199.66.22          2403872.0      2403872.0
July17_0454   192.199.66.23;192.199.66.24          2150347.0      2150347.0
July09_1416   192.199.66.17;192.199.66.18          1887469.0      1887469.0
July08_2209   192.199.66.21;192.199.66.22          1693688.0      1693688.0
July17_0200   192.199.66.23;192.199.66.24          1597102.0      1597102.0
July17_1655   192.199.66.19;192.199.66.20          1567615.0      1567615.0
July11_0400   192.199.66.19;192.199.66.20          1373801.0      1373801.0
July16_1343   192.199.66.17;192.199.66.18          1356319.0      1356319.0
July07_2001   192.199.66.21;192.199.66.22          6774340.0      1354868.0
July10_2006   192.199.66.23;192.199.66.24         10787174.0      1348396.8
July04_1431   192.199.66.19;192.199.66.20          5180228.0      1295057.0
July17_0454   192.199.66.19;192.199.66.20          2579203.0      1289601.5
July09_1901   192.199.66.19;192.199.66.20          1260900.0      1260900.0
July11_0400   192.199.66.23;192.199.66.24          1255291.0      1255291.0
July08_1437   192.199.66.19;192.199.66.20          2439786.0      1219893.0
July17_0457   192.199.66.23;192.199.66.24          1177614.0      1177614.0
July04_0906   192.199.66.23;192.199.66.24          1153969.0      1153969.0
July18_0123   192.199.66.17;192.199.66.18          1153542.0      1153542.0
July17_1957   192.199.66.21;192.199.66.22          1112349.0      1112349.0
July11_0958   192.199.66.23;192.199.66.24          4421561.0      1105390.3

50 LONGEST CELL WAITS IN ASH HISTORY ORDERED BY WAIT TIME

APPROACH: These are the top 50 individual cell waits in ASH
history in wait time order.

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
07/July/18 08:01:45.575 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 2891233531      32768     4381405
16/July/18 02:55:09.946 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 2279513079      32768     3489839
19/JULY/18 08:24:30.057 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3835758527      32768     3027146
16/July/18 02:56:30.076 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2994219
14/July/18 11:07:27.489 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  185394440      32768     2583166
16/July/18 02:56:30.076 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2572340
09/July/18 10:55:35.129 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2524157
09/July/18 10:30:37.201 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2511210
09/July/18 07:04:57.612 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3835758527      32768     2508541
08/July/18 05:17:56.113 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213  842883291      32768     2504764
15/July/18 02:23:59.797 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3835758527      32768     2498919
09/July/18 07:15:15.200 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1122005524      32768     2496073
16/July/18 10:01:28.060 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     2481597
08/July/18 05:17:56.113 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213  842883291      32768     2481323
09/July/18 11:37:22.166 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2478192
16/July/18 12:18:57.851 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 2984361190      32768     2476915
16/July/18 03:00:25.122 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2474114
16/July/18 07:54:01.716 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  758773684      32768     2473662
16/July/18 08:15:35.001 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2473365
15/July/18 01:42:56.817 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2256514768      32768     2472463
08/July/18 05:38:29.082 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 3704102872      32768     2466691
17/July/18 02:28:27.460 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  185394440      32768     2464447
09/July/18 11:07:27.164 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 1122005524      32768     2463370
17/July/18 02:43:03.640 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2460091
16/July/18 03:00:45.182 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133  744236512      32768     2459368
16/July/18 07:37:59.702 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133  744236512      32768     2457007
14/July/18 11:07:25.301 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 4016617326      32768     2453596
17/July/18 02:40:02.464 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131   91591739      32768     2449781
09/July/18 02:20:37.918 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2449762
11/July/18 04:09:24.827 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3409456992      32768     2449147
14/July/18 09:45:22.608 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     2447554
16/July/18 10:52:07.321 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 3958490174      32768     2444207
14/July/18 04:49:22.691 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2441217
04/July/18 02:31:23.455 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2441112
14/July/18 10:03:34.449 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     2440465
04/July/18 03:07:31.342 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     2438825
14/July/18 03:32:54.840 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2437829
17/July/18 02:38:52.304 AM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2263141931      32768     2437640
16/July/18 02:29:45.206 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2437155
16/July/18 08:15:35.001 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2367494631      32768     2436056
16/July/18 10:38:49.813 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2435774
09/July/18 08:12:07.171 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  185394440      32768     2435748
15/July/18 02:44:02.345 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1233700156      32768     2435230
16/July/18 07:38:55.929 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440   24718082      32768     2431597
16/July/18 10:39:39.873 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2430666
10/July/18 03:01:42.766 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2430197
15/July/18 06:11:20.764 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2429871
16/July/18 10:16:11.295 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1019816052      32768     2429717
08/July/18 05:38:19.072 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2263141931      32768     2429692
16/July/18 10:01:38.100 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2428275

100 LONGEST CELL WAITS IN ASH HISTORY ORDERED BY SAMPLE TIME

APPROACH: These are the top 100 individual cell waits in ASH
history in sample time order.

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
04/July/18 02:30:43.598 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1374369774      32768     2424167
04/July/18 03:07:31.342 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     2438825
04/July/18 10:29:07.175 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2417864
04/July/18 10:29:47.246 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 1233700156      32768     2339031
04/July/18 02:31:23.455 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2441112
07/July/18 08:01:45.575 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 2891233531      32768     4381405
08/July/18 01:36:44.865 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  185394440      32768     2420689
08/July/18 01:36:44.865 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     2330502
08/July/18 01:36:53.062 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 1783777120      32768     2420230
08/July/18 04:23:25.390 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     2410974
08/July/18 04:34:13.379 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2256514768      32768     2414668
08/July/18 05:17:46.103 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 1783777120      32768     2319802
08/July/18 05:17:56.113 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213  842883291      32768     2481323
08/July/18 05:17:56.113 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213  842883291      32768     2504764
08/July/18 05:38:19.072 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2263141931      32768     2429692
08/July/18 05:38:29.082 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 3704102872      32768     2466691
08/July/18 05:38:29.082 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 3616317741      32768     2325172
09/July/18 10:13:18.007 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2418070
09/July/18 10:13:18.007 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1404181515      32768     2293017
09/July/18 10:13:23.693 AM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2263141931      32768     2355873
09/July/18 10:13:28.017 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  931391314      32768     2359674
09/July/18 10:30:37.201 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2511210
09/July/18 10:55:35.129 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2524157
09/July/18 10:55:35.129 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2383155
09/July/18 11:07:27.164 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 1122005524      32768     2463370
09/July/18 11:37:22.166 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2478192
09/July/18 01:57:19.921 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1467922874      32768     2337878
09/July/18 02:20:37.918 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2449762
09/July/18 06:51:04.415 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 3266942716      32768     2303697
09/July/18 07:04:57.612 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3835758527      32768     2508541
09/July/18 07:15:15.200 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1122005524      32768     2496073
09/July/18 08:12:07.171 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  185394440      32768     2435748
10/July/18 03:01:42.766 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2430197
10/July/18 01:38:20.572 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1122206058      32768     2396284
10/July/18 08:06:28.912 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 3616317741      32768     2425901
10/July/18 08:06:37.777 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 3704102872      32768     2423546
11/July/18 04:09:24.827 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 3409456992      32768     2449147
14/July/18 09:45:22.608 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1374369774      32768     2447554
14/July/18 10:03:34.449 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     2440465
14/July/18 10:03:54.460 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     2401321
14/July/18 11:07:25.301 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 4016617326      32768     2453596
14/July/18 11:07:27.489 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  185394440      32768     2583166
14/July/18 01:03:23.248 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3958490174      32768     2314095
14/July/18 03:32:54.840 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2437829
14/July/18 04:49:22.691 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2441217
15/July/18 02:23:59.797 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3835758527      32768     2498919
15/July/18 02:28:01.245 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 2984361190      32768     2418245
15/July/18 01:42:26.787 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2263141931      32768     2418862
15/July/18 01:42:56.817 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2256514768      32768     2472463
15/July/18 02:44:02.345 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1233700156      32768     2435230
15/July/18 05:23:22.759 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1404181515      32768     2424663
15/July/18 06:11:20.764 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2429871
15/July/18 06:12:20.859 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1340519488      32768     2299402
16/July/18 02:29:45.206 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2437155
16/July/18 02:31:14.605 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  744236512      32768     2425509
16/July/18 02:35:59.882 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3958490174      32768     2424458
16/July/18 02:55:09.946 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 2279513079      32768     2417036

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
16/July/18 02:55:09.946 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 2279513079      32768     3489839
16/July/18 02:56:30.076 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2994219
16/July/18 02:56:30.076 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2572340
16/July/18 03:00:25.122 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2474114
16/July/18 03:00:45.182 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133  744236512      32768     2459368
16/July/18 03:01:07.779 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  870191132      32768     2338236
16/July/18 09:36:02.169 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1122005524      32768     2395522
16/July/18 10:01:28.060 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     2481597
16/July/18 10:01:38.100 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2428275
16/July/18 10:16:11.295 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1019816052      32768     2429717
16/July/18 10:38:49.813 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 2342661109      32768     2435774
16/July/18 10:39:39.873 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2430666
16/July/18 10:39:39.873 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2414021
16/July/18 10:52:07.321 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 3958490174      32768     2444207
16/July/18 11:47:58.536 AM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2381222600      32768     2422526
16/July/18 12:18:57.851 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 2984361190      32768     2476915
16/July/18 05:01:01.612 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440   24718082      32768     2351090
16/July/18 06:55:13.320 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2375834
16/July/18 06:55:13.320 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2414035
16/July/18 06:56:23.443 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3718979034      32768     2322248
16/July/18 07:37:59.702 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133  744236512      32768     2457007
16/July/18 07:37:59.702 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133  744236512      32768     2417497
16/July/18 07:38:55.929 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440   24718082      32768     2431597
16/July/18 07:54:01.716 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  758773684      32768     2473662
16/July/18 08:15:35.001 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2419493
16/July/18 08:15:35.001 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2367494631      32768     2436056
16/July/18 08:15:35.001 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131  280877291      32768     2473365
17/July/18 02:03:05.401 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133  744236512      32768     2425350
17/July/18 02:38:22.270 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2344538
17/July/18 02:38:49.166 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     2423420
17/July/18 02:38:49.166 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3256856773      32768     2419184
17/July/18 02:38:52.304 AM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213 2263141931      32768     2437640
17/July/18 02:38:59.176 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1433462179      32768     2415119
17/July/18 02:40:02.464 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131   91591739      32768     2449781
17/July/18 02:43:03.640 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2460091
17/July/18 04:58:17.697 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 4112515153      32768     2403872
17/July/18 02:24:05.496 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2381222600      32768     2420104
17/July/18 02:28:27.460 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  185394440      32768     2464447
17/July/18 04:39:42.871 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131   91591739      32768     2297049
19/JULY/18 08:24:30.057 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3835758527      32768     3027146
19/JULY/18 09:55:04.470 AM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1340519488      32768     2363513
19/JULY/18 04:38:55.395 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1317632796      32768     2304803
19/JULY/18 08:04:36.984 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     2388370

AWR CELL DISK UTILIZATION

APPROACH: This query only works in 12.1 and above.  This is looking
in the AWR history tables to look at cell disk utilization for some
of the worst periods.  Top 100 disk utils.
DISK_UTILIZATION_SUM: Sum of the per-minute disk utilization metrics.
IO_REQUESTS_SUM: Sum of the per-minute IOPs.
IO_MB_SUM: Sum of the per-minute I/O metrics, in megabytes per second.

   SNAP_ID BEGIN               END                 CELL_NAME                      DISK_NAME                      DISK_UTILIZATION_SUM IO_REQUESTS_SUM  IO_MB_SUM
---------- ------------------- ------------------- ------------------------------ ------------------------------ -------------------- --------------- ----------
     22669 July13_1300          July13_1330          192.199.66.17;192.199.66.18    /dev/sda                                       1977           11302       1938
     22669 July13_1300          July13_1330          192.199.66.17;192.199.66.18    /dev/sdb                                       1977           11302       1938
     22669 July13_1300          July13_1330          192.199.66.17;192.199.66.18    /dev/sdm                                       1977           11302       1938
     22669 July13_1300          July13_1330          192.199.66.19;192.199.66.20    /dev/sda                                       1926           11343       1979
     22669 July13_1300          July13_1330          192.199.66.19;192.199.66.20    /dev/sdb                                       1926           11343       1979
     22669 July13_1300          July13_1330          192.199.66.19;192.199.66.20    /dev/sdm                                       1926           11343       1979
     22669 July13_1300          July13_1330          192.199.66.21;192.199.66.22    /dev/sda                                       1966           11227       1945
     22669 July13_1300          July13_1330          192.199.66.21;192.199.66.22    /dev/sdb                                       1966           11227       1945
     22669 July13_1300          July13_1330          192.199.66.21;192.199.66.22    /dev/sdm                                       1966           11227       1945
     22669 July13_1300          July13_1330          192.199.66.23;192.199.66.24    /dev/sda                                       1888           11353       1920
     22669 July13_1300          July13_1330          192.199.66.23;192.199.66.24    /dev/sdb                                       1888           11353       1920
     22669 July13_1300          July13_1330          192.199.66.23;192.199.66.24    /dev/sdm                                       1888           11353       1920
     22827 July16_2000          July16_2030          192.199.66.17;192.199.66.18    /dev/sda                                       2068           11683       1428
     22827 July16_2000          July16_2030          192.199.66.17;192.199.66.18    /dev/sdb                                       2068           11683       1428
     22827 July16_2000          July16_2030          192.199.66.17;192.199.66.18    /dev/sdm                                       2068           11683       1428
     22827 July16_2000          July16_2030          192.199.66.19;192.199.66.20    /dev/sda                                       2089           11588       1443
     22827 July16_2000          July16_2030          192.199.66.19;192.199.66.20    /dev/sdb                                       2089           11588       1443
     22827 July16_2000          July16_2030          192.199.66.19;192.199.66.20    /dev/sdm                                       2089           11588       1443
     22827 July16_2000          July16_2030          192.199.66.21;192.199.66.22    /dev/sda                                       1975           11095       1419
     22827 July16_2000          July16_2030          192.199.66.21;192.199.66.22    /dev/sdb                                       1975           11095       1419
     22827 July16_2000          July16_2030          192.199.66.21;192.199.66.22    /dev/sdm                                       1975           11095       1419
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    /dev/sda                                       3808            9676       3421
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    /dev/sdb                                       3808            9676       3421
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    /dev/sdm                                       3808            9676       3421
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_00_monkeynode11_adm                        1929            4811       1713
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_01_monkeynode11_adm                        1879            4623       1703
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_02_monkeynode11_adm                        1793            5186       1674
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_03_monkeynode11_adm                        1808            4887       1633
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_04_monkeynode11_adm                        1923            4612       1719
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_05_monkeynode11_adm                        1898            4812       1750
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_06_monkeynode11_adm                        1887            4748       1677
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_07_monkeynode11_adm                        1905            4917       1720
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_08_monkeynode11_adm                        1792            4997       1687
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_09_monkeynode11_adm                        1623            4926       1501
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_10_monkeynode11_adm                        1921            4466       1739
     22875 July17_2000          July17_2030          192.199.66.17;192.199.66.18    CD_11_monkeynode11_adm                        1949            4469       1723
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    /dev/sda                                       4075           11529       5240
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    /dev/sdb                                       4075           11529       5240
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    /dev/sdm                                       4075           11529       5240
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_00_monkeynode12_adm                        2033            5673       2614
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_01_monkeynode12_adm                        2040            5626       2622
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_02_monkeynode12_adm                        1942            5636       2452
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_03_monkeynode12_adm                        1897            5623       2351
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_04_monkeynode12_adm                        2051            5586       2591
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_05_monkeynode12_adm                        1862            5838       2297
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_06_monkeynode12_adm                        2058            5355       2581
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_07_monkeynode12_adm                        2063            5668       2592
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_08_monkeynode12_adm                        2076            5506       2676
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_09_monkeynode12_adm                        2066            5829       2619
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_10_monkeynode12_adm                        2046            5322       2562
     22875 July17_2000          July17_2030          192.199.66.19;192.199.66.20    CD_11_monkeynode12_adm                        1972            5429       2487
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    /dev/sda                                       3710            9738       3413
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    /dev/sdb                                       3710            9738       3413
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    /dev/sdm                                       3710            9738       3413
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_00_monkeynode13_adm                        1874            4751       1752
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_01_monkeynode13_adm                        1835            4755       1661
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_02_monkeynode13_adm                        1772            5021       1623

   SNAP_ID BEGIN               END                 CELL_NAME                      DISK_NAME                      DISK_UTILIZATION_SUM IO_REQUESTS_SUM  IO_MB_SUM
---------- ------------------- ------------------- ------------------------------ ------------------------------ -------------------- --------------- ----------
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_03_monkeynode13_adm                        1801            4860       1627
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_04_monkeynode13_adm                        1729            4834       1603
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_05_monkeynode13_adm                        2094            6325       2698
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_06_monkeynode13_adm                        1804            4935       1633
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_07_monkeynode13_adm                        1943            4356       1757
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_08_monkeynode13_adm                        1870            4385       1673
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_09_monkeynode13_adm                        1690            4848       1525
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_10_monkeynode13_adm                        1908            5668       1679
     22875 July17_2000          July17_2030          192.199.66.21;192.199.66.22    CD_11_monkeynode13_adm                        1858            4699       1681
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    /dev/sda                                       3677            9714       3381
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    /dev/sdb                                       3677            9714       3381
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    /dev/sdm                                       3677            9714       3381
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_00_monkeynode14_adm                        1853            4752       1685
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_01_monkeynode14_adm                        1822            4750       1693
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_02_monkeynode14_adm                        1874            5117       1725
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_03_monkeynode14_adm                        1813            4593       1665
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_04_monkeynode14_adm                        1810            4736       1637
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_05_monkeynode14_adm                        1869            4575       1717
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_06_monkeynode14_adm                        1789            4769       1669
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_07_monkeynode14_adm                        1840            4634       1734
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_08_monkeynode14_adm                        1838            4480       1666
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_09_monkeynode14_adm                        1753            4656       1641
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_10_monkeynode14_adm                        1729            5019       1574
     22875 July17_2000          July17_2030          192.199.66.23;192.199.66.24    CD_11_monkeynode14_adm                        1849            4579       1710
     22876 July17_2030          July17_2100          192.199.66.17;192.199.66.18    /dev/sda                                       2658           10231       2347
     22876 July17_2030          July17_2100          192.199.66.17;192.199.66.18    /dev/sdb                                       2658           10231       2347
     22876 July17_2030          July17_2100          192.199.66.17;192.199.66.18    /dev/sdm                                       2658           10231       2347
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    /dev/sda                                       3156           12112       4142
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    /dev/sdb                                       3156           12112       4142
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    /dev/sdm                                       3156           12112       4142
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    CD_01_monkeynode12_adm                        1602            5917       2090
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    CD_04_monkeynode12_adm                        1633            5773       2100
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    CD_07_monkeynode12_adm                        1609            5877       2073
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    CD_08_monkeynode12_adm                        1632            5778       2088
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    CD_09_monkeynode12_adm                        1602            6016       2066
     22876 July17_2030          July17_2100          192.199.66.19;192.199.66.20    CD_10_monkeynode12_adm                        1594            5964       2040
     22876 July17_2030          July17_2100          192.199.66.21;192.199.66.22    /dev/sda                                       2449           10793       2281
     22876 July17_2030          July17_2100          192.199.66.21;192.199.66.22    /dev/sdb                                       2449           10793       2281
     22876 July17_2030          July17_2100          192.199.66.21;192.199.66.22    /dev/sdm                                       2449           10793       2281
     22876 July17_2030          July17_2100          192.199.66.21;192.199.66.22    CD_05_monkeynode13_adm                        1569            6433       2025
     22876 July17_2030          July17_2100          192.199.66.23;192.199.66.24    /dev/sda                                       2657           10505       2409
     22876 July17_2030          July17_2100          192.199.66.23;192.199.66.24    /dev/sdb                                       2657           10505       2409
     22876 July17_2030          July17_2100          192.199.66.23;192.199.66.24    /dev/sdm                                       2657           10505       2409

DB_ID_FOR_CURRENT_DB
--------------------
          2490671309

CELL THREAD HISTORY - LAST FEW MINUTES

This query only works in 12.1 and above.

  COUNT(*) SQL_ID        CELL_NAME                      JOB_TYPE                         DATABASE_ID INSTANCE_ID
---------- ------------- ------------------------------ -------------------------------- ----------- -----------
      1598               192.199.66.19;192.199.66.20    UnidentifiedJob                            0           0
      1555               192.199.66.23;192.199.66.24    UnidentifiedJob                            0           0
      1237               192.199.66.17;192.199.66.18    UnidentifiedJob                            0           0
      1148               192.199.66.21;192.199.66.22    UnidentifiedJob                            0           0
       834               192.199.66.21;192.199.66.22    NetworkPoll                                0           0
       824               192.199.66.17;192.199.66.18    NetworkPoll                                0           0
       804               192.199.66.19;192.199.66.20    NetworkPoll                                0           0
       804               192.199.66.23;192.199.66.24    NetworkPoll                                0           0
       417               192.199.66.21;192.199.66.22    OCL System Message Thread                  0           0
       417               192.199.66.21;192.199.66.22    Storage index eviction                     0           0
       412               192.199.66.17;192.199.66.18    OCL System Message Thread                  0           0
       412               192.199.66.17;192.199.66.18    Storage index eviction                     0           0
       402               192.199.66.19;192.199.66.20    OCL System Message Thread                  0           0
       402               192.199.66.19;192.199.66.20    Storage index eviction                     0           0
       402               192.199.66.23;192.199.66.24    OCL System Message Thread                  0           0
       402               192.199.66.23;192.199.66.24    Storage index eviction                     0           0
       332               192.199.66.23;192.199.66.24    NetworkRead                                0           0
       281               192.199.66.21;192.199.66.22    NetworkRead                                0           0
       254               192.199.66.17;192.199.66.18    NetworkRead                                0           0
       193 9ujkk29vay3bz 192.199.66.23;192.199.66.24    PredicateDiskRead                  584354840           1
       168 9ujkk29vay3bz 192.199.66.17;192.199.66.18    PredicateDiskRead                  584354840           1
       165 9ujkk29vay3bz 192.199.66.19;192.199.66.20    PredicateDiskRead                  584354840           1
       152 9ujkk29vay3bz 192.199.66.21;192.199.66.22    PredicateDiskRead                  584354840           1
       135               192.199.66.19;192.199.66.20    NetworkRead                                0           0
       112               192.199.66.17;192.199.66.18    CacheGet                                   0           0
        86               192.199.66.19;192.199.66.20    CacheGet                                   0           0
        83               192.199.66.21;192.199.66.22    CacheGet                                   0           0
        66               192.199.66.23;192.199.66.24    CacheGet                                   0           0
        55 aym6pqm5uzd90 192.199.66.17;192.199.66.18    PredicateDiskRead                  584354840           1
        52 aym6pqm5uzd90 192.199.66.21;192.199.66.22    PredicateDiskRead                  584354840           1
        51 aym6pqm5uzd90 192.199.66.23;192.199.66.24    PredicateDiskRead                  584354840           1
        50 oj2tdhpjgnvus 192.199.66.23;192.199.66.24    CacheGet                           584354840           1
        47 aym6pqm5uzd90 192.199.66.19;192.199.66.20    PredicateDiskRead                  584354840           1
        37 oj2tdhpjgnvus 192.199.66.17;192.199.66.18    CacheGet                           584354840           1
        34 oj2tdhpjgnvus 192.199.66.19;192.199.66.20    CacheGet                           584354840           1
        29 oj2tdhpjgnvus 192.199.66.21;192.199.66.22    CacheGet                           584354840           1
        18               192.199.66.17;192.199.66.18    PredicateFilter                            0           0
        17               192.199.66.23;192.199.66.24    PredicateFilter                            0           0
        16               192.199.66.19;192.199.66.20    PredicateFilter                            0           0
        16 221fz5z4guyxu 192.199.66.17;192.199.66.18    PredicateDiskRead                 1377831170           1
        15               192.199.66.21;192.199.66.22    PredicateFilter                            0           0
        15               192.199.66.23;192.199.66.24    CachePut                                   0           0
        14 c10mc2tzkhbwb 192.199.66.23;192.199.66.24    PredicateDiskRead                 1377831170           1
        13               192.199.66.23;192.199.66.24    CachePut                          1377831170           1
        12               192.199.66.21;192.199.66.22    CachePut                                   0           0
        11               192.199.66.17;192.199.66.18    CachePut                                   0           0
        10               192.199.66.19;192.199.66.20    CachePut                                   0           0
        10               192.199.66.19;192.199.66.20    PredicateCacheGet                          0           0
         9               192.199.66.21;192.199.66.22    CachePut                           584354840           1
         9 221fz5z4guyxu 192.199.66.19;192.199.66.20    PredicateDiskRead                 1377831170           1

CELL CONFIG


CELLNAME
----------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFVAL
----------------------------------------------------------------------------------------------------------------------------------------------------------------
192.199.66.17;192.199.66.18


  1.0
  1526659672886
  
  
    monkeynode11_adm
    remoteLoginEpdiled
    normal
    OSS_12.2.1.1.6_LINUX.X64_180125.1
    40/40
    7
    text
    8/8
    normal
    WriteThrough
    1614NM70A0
    2
    ib0
    ib1
    0.016666666666666666
    192.199.66.17/22
    192.199.66.18/22
    4.1.12-94.7.8.el6uek.x86_64
    off
    Oracle Corporation ORACLE SERVER X6-2L High Capacity
    126
    7
    mail
    critical,warning,clear
    
    2/2
    normal
    success
    12.2.1.1.6.180125.1
    cell-12.2.1.1.6_LINUX.X64_180125.1-1.x86_64
    27351071
    12.1.2.3.6.170713
    "PRODA Exadata Cell 01"
    monkeynode11-adm
    11.20.238.25
    pdiPD_pdiSTRATUS.droid@pdi.com.PD
    FALSE
     host=11.47.4.195,port=162,community=public,type=asr,asrmPort=16161
    online
    18.0
    normal
    22 days, 1:03
    normal
  


192.199.66.19;192.199.66.20


CELLNAME
----------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFVAL
----------------------------------------------------------------------------------------------------------------------------------------------------------------

  1.0
  1526662533109
  
  
    monkeynode12_adm
    remoteLoginEpdiled
    normal
    OSS_12.2.1.1.6_LINUX.X64_180125.1
    40/40
    7
    text
    8/8
    normal
    WriteThrough
    1614NM705Y
    2
    ib0
    ib1
    0.0
    192.199.66.19/22
    192.199.66.20/22
    4.1.12-94.7.8.el6uek.x86_64
    off
    Oracle Corporation ORACLE SERVER X6-2L High Capacity
    126
    7
    mail
    critical,warning,clear
    
    2/2
    normal
    success
    12.2.1.1.6.180125.1
    cell-12.2.1.1.6_LINUX.X64_180125.1-1.x86_64
    27351071
    12.1.2.3.6.170713
    "PRODA Exadata Cell 02"
    monkeynode12-adm
    11.20.238.25
    pdiPD_pdiSTRATUS.droid@pdi.com.PD
     host=11.47.4.195,port=162,community=public,type=asr,asrmPort=16161
    online
    18.0
    normal
    22 days, 1:03
    normal
  


192.199.66.21;192.199.66.22


  1.0
  1526661149874

CELLNAME
----------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFVAL
----------------------------------------------------------------------------------------------------------------------------------------------------------------
  
  
    monkeynode13_adm
    remoteLoginEpdiled
    normal
    OSS_12.2.1.1.6_LINUX.X64_180125.1
    40/40
    7
    text
    8/8
    normal
    WriteThrough
    9717PD519T
    2
    ib0
    ib1
    0.0
    192.199.66.21/22
    192.199.66.22/22
    4.1.12-94.7.8.el6uek.x86_64
    off
    Oracle Corporation ORACLE SERVER X6-2L High Capacity
    126
    7
    mail
    critical,warning,clear
    
    2/2
    normal
    success
    12.2.1.1.6.180125.1
    cell-12.2.1.1.6_LINUX.X64_180125.1-1.x86_64
    27351071
    12.1.2.3.6.170713
    "PRODA Exadata Cell 03"
    monkeynode13-adm
    11.20.238.25
    pdiPD_pdiSTRATUS.droid@pdi.com.PD
     host=11.47.4.195,port=162,community=public,type=asr,asrmPort=16161
    online
    18.0
    normal
    21 days, 23:03
    normal
  


192.199.66.23;192.199.66.24


  1.0
  1526660912516
  
  
    monkeynode14_adm

CELLNAME
----------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFVAL
----------------------------------------------------------------------------------------------------------------------------------------------------------------
    remoteLoginEpdiled
    normal
    OSS_12.2.1.1.6_LINUX.X64_180125.1
    40/40
    7
    text
    8/8
    normal
    WriteThrough
    1635NM70Y9
    2
    ib0
    ib1
    0.016666666666666666
    192.199.66.23/22
    192.199.66.24/22
    4.1.12-94.7.8.el6uek.x86_64
    off
    Oracle Corporation ORACLE SERVER X6-2L High Capacity
    126
    7
    mail
    critical,warning,clear
    
    2/2
    normal
    success
    12.2.1.1.6.180125.1
    cell-12.2.1.1.6_LINUX.X64_180125.1-1.x86_64
    27351071
    12.1.2.3.6.170713
    "PRODA Exadata Cell 04"
    monkeynode14-adm
    11.20.238.25
    pdiPD_pdiSTRATUS.droid@pdi.com.PD
     host=11.47.4.195,port=162,community=public,type=asr,asrmPort=16161
    online
    17.0
    normal
    21 days, 22:09
    normal
  



IORM CONFIG


CELLNAME
----------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFVAL
----------------------------------------------------------------------------------------------------------------------------------------------------------------
192.199.66.17;192.199.66.18


  1.0
  1526659674006
  
  
    monkeynode11_adm_IORMPLAN
    
    
    basic
    active
  


192.199.66.19;192.199.66.20


  1.0
  1526662534081
  
  
    monkeynode12_adm_IORMPLAN
    
    
    basic
    active
  


192.199.66.21;192.199.66.22


  1.0
  1526661150877
  
  
    monkeynode13_adm_IORMPLAN
    
    
    basic
    active
  


192.199.66.23;192.199.66.24


  1.0
  1526660913655
  
  
    monkeynode14_adm_IORMPLAN
    
    

CELLNAME
----------------------------------------------------------------------------------------------------------------------------------------------------------------
CONFVAL
----------------------------------------------------------------------------------------------------------------------------------------------------------------
    basic
    active
  



TIME
-----------------------
July19 02:57:21
1133011440 1122005524      32768     1719451
19/JULY/18 04:39:45.763 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131   91591739      32768     1669777
19/JULY/18 04:39:53.783 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     1606474
19/JULY/18 04:47:20.930 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 1467922874      32768     1598316
19/JULY/18 04:47:21.101 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 1467922874      32768     1723089
19/JULY/18 04:48:26.040 PM 192.199.66.23;192.199.66.24          2 cell single block physical read          3802949213  842883291      32768     2359311
19/JULY/18 05:17:36.317 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  125724152      32768     1743703
19/JULY/18 05:23:11.293 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  552535141      32768     1631948
19/JULY/18 05:23:19.727 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131   91591739      32768     1741381
19/JULY/18 06:23:57.158 PM 192.199.66.19;192.199.66.20          1 cell single block physical read          1318032131 1374369774      32768     1769798
19/JULY/18 08:03:38.870 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213 2701596410      32768     1622627
19/JULY/18 08:03:38.870 PM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440 2251483371      32768     2032547
19/JULY/18 08:03:39.874 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  474454945      32768     2024898
19/JULY/18 08:03:39.874 PM 192.199.66.23;192.199.66.24          2 cell smart incremental backup            3802949213          0          0     1775912
19/JULY/18 08:04:18.964 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 4112515153      32768     3333391
19/JULY/18 08:04:19.964 PM 192.199.66.21;192.199.66.22          2 cell smart incremental backup            2990700133          0          0     5560085
19/JULY/18 08:04:28.984 PM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 3409456992      32768     2098081
19/JULY/18 08:04:36.984 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     2388370
19/JULY/18 08:07:04.178 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1233700156      32768     2102692
19/JULY/18 08:09:11.705 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133  870191132      32768     1753415
19/JULY/18 10:13:12.946 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3256856773      32768     1777250

ASH HISTORY CELL PERFORMANCE SUMMARY

This query will look at the average cell wait times for each cell in ASH

CELL_PATH                      TOTAL_WAIT_TIME  AVG_WAIT_TIME
------------------------------ --------------- --------------
192.199.66.19;192.199.66.20       4320370317.0       144803.9
192.199.66.17;192.199.66.18       3720276327.0       126342.3
192.199.66.21;192.199.66.22       3302364789.0       116198.6
192.199.66.23;192.199.66.24       2214253857.0        86457.1

20 WORST CELL PERFORMANCE MINUTES IN ASH HISTORY:

APPROACH: These are the minutes where the avg cell perf time
was the highest.  See which cell had the longest waits and
during what time minute.

MINUTE       CELL_PATH                      TOTAL_WAIT_TIME  AVG_WAIT_TIME
------------ ------------------------------ --------------- --------------
July17_0458   192.199.66.21;192.199.66.22          2403872.0      2403872.0
July17_0454   192.199.66.23;192.199.66.24          2150347.0      2150347.0
July09_1416   192.199.66.17;192.199.66.18          1887469.0      1887469.0
July08_2209   192.199.66.21;192.199.66.22          1693688.0      1693688.0
July17_0200   192.199.66.23;192.199.66.24          1597102.0      1597102.0
July17_1655   192.199.66.19;192.199.66.20          1567615.0      1567615.0
July11_0400   192.199.66.19;192.199.66.20          1373801.0      1373801.0
July16_1343   192.199.66.17;192.199.66.18          1356319.0      1356319.0
July07_2001   192.199.66.21;192.199.66.22          6774340.0      1354868.0
July10_2006   192.199.66.23;192.199.66.24         10787174.0      1348396.8
July04_1431   192.199.66.19;192.199.66.20          5180228.0      1295057.0
July17_0454   192.199.66.19;192.199.66.20          2579203.0      1289601.5
July09_1901   192.199.66.19;192.199.66.20          1260900.0      1260900.0
July11_0400   192.199.66.23;192.199.66.24          1255291.0      1255291.0
July08_1437   192.199.66.19;192.199.66.20          2439786.0      1219893.0
July17_0457   192.199.66.23;192.199.66.24          1177614.0      1177614.0
July04_0906   192.199.66.23;192.199.66.24          1153969.0      1153969.0
July18_0123   192.199.66.17;192.199.66.18          1153542.0      1153542.0
July17_1957   192.199.66.21;192.199.66.22          1112349.0      1112349.0
July11_0958   192.199.66.23;192.199.66.24          4421561.0      1105390.3

50 LONGEST CELL WAITS IN ASH HISTORY ORDERED BY WAIT TIME

APPROACH: These are the top 50 individual cell waits in ASH
history in wait time order.

SAMPLE_TIME               CELL_PATH                      INST_ID EVENT                                     CELLHASH#  DISKHASH#      BYTES TIME_WAITED
------------------------- ------------------------------ ------- ---------------------------------------- ---------- ---------- ---------- -----------
07/July/18 08:01:45.575 PM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 2891233531      32768     4381405
16/July/18 02:55:09.946 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 2279513079      32768     3489839
19/JULY/18 08:24:30.057 AM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3835758527      32768     3027146
16/July/18 02:56:30.076 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2994219
14/July/18 11:07:27.489 AM 192.199.66.17;192.199.66.18          1 cell single block physical read          1133011440  185394440      32768     2583166
16/July/18 02:56:30.076 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2572340
09/July/18 10:55:35.129 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3530077886      32768     2524157
09/July/18 10:30:37.201 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1317632796      32768     2511210
09/July/18 07:04:57.612 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 3835758527      32768     2508541
08/July/18 05:17:56.113 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213  842883291      32768     2504764
15/July/18 02:23:59.797 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3835758527      32768     2498919
09/July/18 07:15:15.200 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 1122005524      32768     2496073
16/July/18 10:01:28.060 AM 192.199.66.21;192.199.66.22          1 cell single block physical read          2990700133 3770061594      32768     2481597
08/July/18 05:17:56.113 PM 192.199.66.23;192.199.66.24          1 cell single block physical read          3802949213  842883291      32768     2481323
09/July/18 11:37:22.166 AM 192.199.66.19;192.199.66.20          2 cell single block physical read          1318032131 1019816052      32768     2478192
16/July/18 12:18:57.851 PM 192.199.66.21;192.199.66.22          2 cell single block physical read          2990700133 2984361190      32768     2476915
16/July/18 03:00:25.122 AM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440 3266942716      32768     2474114
16/July/18 07:54:01.716 PM 192.199.66.17;192.199.66.18          2 cell single block physical read          1133011440  758773684      32768     2473662
16/July/18 08:15:35.001 PM 192.199.66.19;192.199.66.20  

Hope It Helps
Prashant Dixit

Advertisements

Posted in Advanced | Tagged: , | 2 Comments »

Want to monitor your Exa Health — Answer is ‘Exachk’ …

Posted by FatDBA on December 30, 2014

Recently while working on one of the test exadata machine came across one of the very useful tool to monitoring exadata system health. Being one of the most popular topics that customers talk about is the monitoring of your exadata health.

The best tool for this is the Exachk (see MOS Doc ID 1070954.1)
This document contains the current Exachk release.

The recommendation or advices for Exachk is to:

1) Run the exachk (at a minimum) quarterly, and after any changes are made to the configuration
2) ALWAYS run the current exachk.  This script is periodically updated/improved upon so it is very important to be current
3) Keep track of any failures to ensure that you can identify any new items that appear in the report
4) A score of 80 or above is a good score for production. It is very rare to have a score that is 99+.

There are also a great whitepaper released in September 2013. This white paper can be referred and downloaded from here.
http://www.oracle.com/technetwork/database/availability/exadata-health-resource-usage-2021227.pdf

Thanks
Prashant Dixit

Posted in Advanced | Tagged: , | 2 Comments »

 
%d bloggers like this: