• mysql -u zenoss -p
  mysql -uroot

  • MySQL prompt for doing SQL queries
  • -u username (zenoss in this case)
  • -p password; without the password written in here, it prompts you for it, so that the pass is obscured

• mtop
  mysqladmin processlist | grep -v Sleep
  show full processlist\G;

  • Query list; running queries in MySQL
  • in mtop, ? gives keys you can press, s = seconds b/w updates
  • use 2nd command to get shell output
  • I extended 2nd line in mysql prompt to get 3rd line (shows entire long query)

• SHOW DATABASES;
  USE events;
  SHOW tables;
  DESCRIBE status;
  SHOW table status;

  • Database schema list
  • Switch to a certain database schema
  • Table list
  • Event console attribute names (metadata)
  • Show status of all tables in current DB (row count, DB engine type)

• mysql events -e "show table status"
  mysql -uroot information_schema -e "select TABLE_NAME, TABLE_ROWS from TABLES where TABLE_SCHEMA = 'events'"

  • Row counts in tables, engine types, Index Info, metadata

• OPTIMIZE table status, history, alert_state, detail, LOG, heartbeat;
  ANALYZE table status, history, alert_state, detail, LOG, heartbeat;

  • Optimize the database
  • Both commands *must* be run in this order to be effective

• DELETE h,j,d
  FROM history h LEFT JOIN j ON h.evid=j.evid LEFT JOIN detail d ON h.evid=d.evid
  WHERE statechange < DATE_SUB(NOW(), INTERVAL 33 DAY);

  • Delete historic events older than 33 days
  • Same as automatic (occurs constantly) in-GUI functionality:
    • Event Manager> Delete Historical Events Older Than (days)

• CREATE INDEX stateChange_idx
  ON history(stateChange);

  • Puts an index on stateChange column
  • Good for when history table deletion (daily parring down) is going slowww
  • Event Manager > Delete Historical Events Older Than (days)

• # As root
  service zenoss stop
  service mysqld stop
  cd /var/lib
  mv mysql oldmysql
  mkdir mysql
  chown mysql:mysql mysql
  service mysqld start
   
  # As zenoss user
  zeneventbuild localhost root '' events 3306
  mysql -uroot -e "grant all privileges on events.* to zenoss@localhost identified by 'zenoss'"
   
  start up zenoss
   
  CHECK tables through mysql prompt
  CHECK UI events console

  • Drops the mysql events database completely
  • Recreates the events DB
  • from chet

• mysqldump -uroot --routines events > events.sql
  mysql -uroot -p #
  mysql> CREATE DATABASE events;
  mysql> \. events.sql
  mysql> GRANT ALL ON events.* TO zenoss@IPADDRESS IDENTIFIED BY 'zenoss';
  mysql> FLUSH PRIVILEGES;

  • Remote mysql database
  • -p only if password is required.
  • replace “IPADDRESS” the IP address of your Zenoss server, to allow it to remotely access the events database
  • You will then need to go into the Event Manager section of the Zenoss web interface and change the database hostname to the IP address or hostname of your remote database server.

• chkconfig --list
  chkconfig mysqld on

  • Start mysqld on boot-up

• SELECT severity, COUNT(severity) FROM history WHERE lastTime>(UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 5 MINUTE))) GROUP BY severity UNION ALL SELECT severity, COUNT(severity) FROM status WHERE lastTime>(UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 5 MINUTE))) GROUP BY severity;

  • Show histogram of last five minutes of events
    • how many events in each category of severities came in?
    • queries both history and status (live events) table
    • example of using lastTime which is oddly a ‘double’ instead of ‘timestamp’ data type

• zenmodeler run -v10 -d applianceincloset.zenoss.loc --collect=zenoss.snmp.HRSWRunMap

  • Processes running, as reported to Zenoss by SNMP
  • runs the modeler daemon on a device, with high verbosity
  • –collect=zenoss.snmp.HRSWRunMap is the modeler plugin that gives you processes

• snmpwalk -v1 -c public applianceincloset.zenoss.loc .1.3.6.1.2.1.25

  • Processes running, as reported by SNMP

• wmic -d99 -U "DOMAIN\Username%Password" //IP "select * from win32_process"

  • Processes running, as reported to Zenoss by WMI
  • run on zenoss machine; wmic exists on windows machines too, but has totally different options
  • -d99 high verbosity output
  • -U user authentication
  • IP device name or IP
  • select statement can be modified; this one gives listing of processes as reported to zenoss by wmi

• select * from win32_process

  • Processes running, as reported on local device by WMI
  • run inside wbemtest.exe
  • currently, only gives numeric ID of processes

• # prowalker.sh
  #!/bin/sh
  #
  # Update the SNMP version, community string and IP address in the command below.
  while [ 1 ]; do
  	STAMP=`date  %Y%m%d%H%M`
      echo "$STAMP: Walking process table.."
      snmpwalk -v2c -c public 127.0.0.1 .1.3.6.1.2.1.25.4 \
          > procwalk.$STAMP
      sleep 60
  done
   
   
  ll procwalk.* | wc -l
  grep -lr "string of process in files" . | wc -l

  • Are processes being reported by snmp agent consistently over time?
  • Basically what you do is run an snmpwalk command every minute for the list of processes on a particular box, and record those to a file. You need to do this for some long amount of time, give it a day, or even to run over a weekend if you have the drive space and patience.
  • You will then be able to count how many files there are (one for each minute).
  • Also, you will count those files to see how many files mention the process of interest.
  • If everything is working as it should, you will get the same number of files for each count (one for each snmpwalk run). If the numbers are different, we know that the agent is not sending the process list correctly each and every time.
  • case 7682

• dmd.ZenUsers.manage_addUser('zenoss-support', 'zen0ss', ['Manager',])
  commit()

  • User add via dmd

• zendmd
   
  from Products.ZenWidgets.ZenossPortlets.ZenossPortlets import portlets
  portlets

  • Portlets registered listing
  • Shows the objects in the zopedb
  • first command is from shell, to get into dmd shell

• dmd.ZenUsers.manage_deleteUsers(["test"])
  commit()

  • delete a zenoss user /object via the DMD
  • To be run in the DMD console, zendmd

• dmd.version

  • Version that zenoss codebase thinks it is at
  • If a migrate / upgrade gets part way through, and errors out, this may show a newer version than it is really on.

• httpd = dmd.getObjByPath("/zport/dmd/Processes/Apache/osProcessClasses/httpd")
  for i in httpd.instances():
      try:
          tmp = i.getDeviceLink("deviceOsDetail")
          print "found %s" % i.id
      except:
          print "/nDELETING %s/n" % i.id
          #comment this in after first run
          httpd.instances._remove(i)

  • Delete Objects that have incorrect process relation links
  • Symptom is when you cannot click on a certain process to see its details and instances
  • Case sent to chet

• for ec in dmd.Events.getSubOrganizers():
      if ec.transform:
          print "=== %s ===" % ec.getOrganizerName()
          print ec.transform
          print
      for i in ec.instances():
          if i.transform:
              print "=== %s/%s ===" % (ec.getOrganizerName(), i.id)
              print i.transform
              print

  • Transform list from a zenoss system
  • Headings represent where the transforms are located, followed by the content of the transform
  • See python version here transform_listing

• for device in dmd.Devices.Network.Switch.getSubDevices():
      print device
      for interface in device.os.interfaces():
          print "   %s" % (interface)
          interface.monitor = False

  • Turn off all interface monitoring
  • Only applies to Devices/Network/Switch
  • dmd run

The zeo database server is separate from the zope web server and needs to be on a different port, usually 8100 by default. Make sure that your zeo is on the proper port (in server/etc/zeo.conf I think) and that your zope instances are configured to access that port (in client1/etc/zope.conf I think).

The FileSystem database, an object database store for zope (ZODB)

• details at http://wiki.zope.org/ZODB/FileStorageBackup

fstest [-v[v]] Data.fs

• verifies low-level integrity

fsrefs [-v] Data.fs

• checks object sanity

• log.debug("message here")

  • Print a debug message to zenhub.log
  • Can be used to debug transforms

• zenhub debug

  • Set zenhub in debug mode
  • That switches it from default to normal and then back again.
  • No restart required.
  • That actually works for all of the daemons in 2.5.1, btw.
  • prints ALL of the event fields that came in, the transform that it’s going to use, and then the resulting stuff that gets written to the database.
  • There’s also an awk script that allows you to *just* see the event stuff, rather than all DEBUG level events. (Kells)

• for pack in /opt/zenoss/packs/*.egg; do zenpack --install $pack; done

  • Reinstall all zenpacks
  • only installs zenpacks in /packs directory

• wget -O /dev/null http://admin:zenoss@localhost:8080/zport/dmd/Devices/Server/Linux/devices/build.zenoss.loc/deleteDevice

  • delete a device from shell
  • output is ignored to /dev/null
  • -O output redirect to file

• curl http://admin:zenoss@train4.zenoss.loc:8080/zport/dmd/Devices/Server/Linux/devices/build.zenoss.loc/deleteDevice

  • delete a device from shell
  • in case you dont have wget

• zenperfsnmp run -v10 -d <device_name_inserted_here> -c --showdeviceresults
  zenwebtx run -v10 -d <device_name_inserted_here>

  • SNMP troubleshooting
  • runs snmp performance daemon one time; not in background / daemon
  • you have to wait until the 5 minute cycle comes around and shows data
  • then hit ctrl C to get out
  • -v10 high verbosity
  • -d specify a device
  • -c cycle over and over
  • –showdeviceresults see the actual raw values coming in
  • if you see: INFO:zen.zenperfsnmp:Configured 0 of 0 devices
  • if you see: INFO:zen.zenperfsnmp:sent 0 OID requests
    • nothing worked, you want >0 OIDs

• zenwinperf run -c -v10 -d <device_name_inserted_here>

  • windows performance troubleshooting
  • -v10 high verbosity
  • -d specify a device
  • details of key log lines and their meanings

• snmpwalk -v1 -c public applianceincloset.loc <oid_inserted_here_if_known>

  • shows snmp data
  • good for snmp troubleshooting
  • will immediately tell you if SNMP is up and working properly
  • -v1 verbosity increased
  • -c community string, this is kind of like an ‘snmp password’, public by default
  • applianceincloset.loc system to run on, could also be an IP address

• dscl . -delete /Users/zenoss ; dscl . -delete /Groups/zenoss

  • for removing hidden user/group, when uninstalling stack install on mac
  • Mac only
  • “.” represents the local netinfo node
  • - delete delete the following Directory Service directory node
  • PS: this command has many uses, you can do remote ldap config with it:
    • dscl -u <admin name> -P <admin password> /LDAPv3/127.0.0.1 -create /Users/username
  • http://forums.zenoss.com/viewtopic.php?t=9019