{"id":83,"date":"2009-09-04T04:41:03","date_gmt":"2009-09-04T09:41:03","guid":{"rendered":"http:\/\/www.hpux.ws\/?p=83"},"modified":"2009-09-04T04:42:08","modified_gmt":"2009-09-04T09:42:08","slug":"q4-crash-dump-analysis","status":"publish","type":"post","link":"http:\/\/www.hpux.ws\/?p=83","title":{"rendered":"Q4 Crash Dump Analysis to Analyze System Dump Files"},"content":{"rendered":"<p>What follows is a document I found on the forums. It can also be found on the docs.hp.com site, but this is a paraphrase, with some extra commentary.<\/p>\n<p>1) You need to have foresight. Before you have a crash you must enable your system to save crash dumps.<\/p>\n<p>2) vi \/etc\/rc.config.d\/savecrash\u00a0\u00a0\u00a0 &#8212; set the first parameter to 1. Now when your system crashes, and some day it probably will you can perform q4 analysis and send the results to HP. I think this document originated within HP. I have one written somewhere on the forums, but his one is better.<\/p>\n<p><img decoding=\"async\" src=\"file:\/\/\/tmp\/moz-screenshot.jpg\" alt=\"\" \/><\/p>\n<p><!-- \t\t@page { margin: 0.79in } \t\tH1 { margin-bottom: 0.04in } \t\tH1.western { font-family: \"Arial\", \"Helvetica\", sans-serif; font-size: 14pt } \t\tH1.cjk { font-family: \"DejaVu Sans\"; font-size: 14pt } \t\tH1.ctl { font-family: \"DejaVu Sans\"; font-size: 12pt; font-weight: normal } \t\tH2 { margin-bottom: 0.04in } \t\tH2.western { font-family: \"Arial\", \"Helvetica\", sans-serif; font-size: 12pt; font-style: italic } \t\tH2.cjk { font-family: \"DejaVu Sans\"; font-size: 12pt; font-style: italic } \t\tH2.ctl { font-family: \"DejaVu Sans\"; font-size: 12pt; font-weight: normal } \t\tP { margin-bottom: 0.08in } \t\tA:link { color: #0000ff } --><\/p>\n<h1>USING Q4 TO ANALYZE SYSTEM DUMP FILES<\/h1>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"> &#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-<\/span><\/span><\/p>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">When a 11.X HP-UX system crashes, it saves a snapshot of RAM in swap and during the reboot, copies it into \/var\/adm\/crash.  Because these files are binary, a utility called \u201cq4\u201d is used to analyze them and create readable text from which the response center can determine the failure cause.<\/span><\/p>\n<h2>============================  STEP 1 ===========================<\/h2>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">Dumps are normally saved to \/var\/adm\/crash.<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">Verify you have a dump to analyze by doing:<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"># ll \/var\/adm\/crash\/cr*<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">You may see:<\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">\/var\/adm\/crash\/crash.0\/INDEX<\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">\/var\/adm\/crash\/crash.0\/vmunix.gz<\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">\/var\/adm\/crash\/crash.0\/image.0.1.gz<\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">\/var\/adm\/crash\/crash.0\/image.0.2.gz<\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">\/var\/adm\/crash\/crash.0\/image.0.3.gz<\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">\/var\/adm\/crash\/crash.0\/image.0.4.gz<\/span><\/p>\n<p style=\"margin-left: 1in; text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">^ your suffix may vary<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">The INDEX file contains and the \/etc\/shutdownlog contains the \u201cpanic\u201d statement. <\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<h2>============================  STEP 2 ===========================<\/h2>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">The following commands must all be run from the dump directory:<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">cd \tto the dump directory    ie:  cd \/var\/adm\/crash\/crash.0<\/span><\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"> ^^^^ ^<\/span><\/span><\/p>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"> your dump dir.<\/span><\/span><\/p>\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"># \t\/usr\/contrib\/bin\/gunzip vmunix.gz<\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"text-indent: 0.25in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">(uncompresses the kernel file \u2013 may already be done)<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"># \tq4prep -p <\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"text-indent: 0.25in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">(ignore the error if this was previously done)<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">Now \ttype: <\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"text-indent: 0.25in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"># q4 -p . <\/span><\/p>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"> ^  Notice this \u2018dot\u2019<\/span><\/span><\/p>\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">This will put you at the q4 utility prompt:  q4&gt;<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">The \tnext command will get you a \u201cfingerprint\u201d of what was going on \ton the system at the time of the failure. <\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">If \tyou are working with an HP RCE at this time, type the following line \tand read the results to him:<\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">trace event 0<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">Otherwise, simply type this next line and continue.<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"text-indent: 0.5in; margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">trace event 0 &gt; trace<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">At \tthe prompt type:  include analyze.pl <\/span><\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"> \\_letter \u201cel\u201d<\/span><\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">At \tthe next prompt type:   run Analyze AU &gt;&gt; ana.out<\/span><\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"margin-bottom: 0in;\">\n<ul>\n<li>\n<p style=\"margin-bottom: 0in;\"><span style=\"color: #000000;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">At \tthe next prompt type:   exit<\/span><\/span><\/p>\n<\/li>\n<\/ul>\n<p style=\"margin-bottom: 0in;\">\n<h2>============================  STEP 3===========================<\/h2>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">Generate a patch list: <\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\"># swlist -l product PH\\*  &gt; patch_list<\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">Using the CALL ID as the subject, email patch_list, ana.out and possibly the trace file and what.out to : <\/span><span style=\"color: #0000ff;\"><span style=\"text-decoration: underline;\"><a href=\"mailto:hpcu@atl.hp.com\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">hpcu@atl.hp.com<\/span><\/a><\/span><\/span><\/p>\n<p style=\"margin-bottom: 0in;\">\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">NOTE: Max 3MB email size<\/span><\/p>\n<p style=\"margin-bottom: 0in;\"><span style=\"font-family: Arial,Helvetica,sans-serif;\">To speed future calls of this nature, open a call with the Response Center and inform them that you will send email with the call ID as the subject.  Then send the ana.out and patch_list file to the email address listed above.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>What follows is a document I found on the forums. It can also be found on the docs.hp.com site, but this is a paraphrase, with some extra commentary. 1) You need to have foresight. Before you have a crash you must enable your system to save crash dumps. 2) vi \/etc\/rc.config.d\/savecrash\u00a0\u00a0\u00a0 &#8212; set the first [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"_kadence_starter_templates_imported_post":false,"footnotes":""},"categories":[8],"tags":[22,6,18,21],"class_list":["post-83","post","type-post","status-publish","format-standard","hentry","category-sysadmin","tag-crash-dump","tag-forumsitrchpcom","tag-hp-ux","tag-q4"],"_links":{"self":[{"href":"http:\/\/www.hpux.ws\/index.php?rest_route=\/wp\/v2\/posts\/83"}],"collection":[{"href":"http:\/\/www.hpux.ws\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.hpux.ws\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.hpux.ws\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.hpux.ws\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=83"}],"version-history":[{"count":2,"href":"http:\/\/www.hpux.ws\/index.php?rest_route=\/wp\/v2\/posts\/83\/revisions"}],"predecessor-version":[{"id":85,"href":"http:\/\/www.hpux.ws\/index.php?rest_route=\/wp\/v2\/posts\/83\/revisions\/85"}],"wp:attachment":[{"href":"http:\/\/www.hpux.ws\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=83"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.hpux.ws\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=83"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.hpux.ws\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=83"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}