technology:programming:github [Nick Yeates -dot- com]

This doc pulled from http://community.zenoss.org/docs/DOC-8495/version/58

There are images and text formating that are missing here. I also saved it locally to dropbox within /work/zenoss/zenpacks/

—

ZENPACK DEVELOPMENT PROCESS VERSION 58 Click to view document history Created on: Sep 16, 2010 5:45 PM by Matt Ray - Last Modified: Dec 21, 2011 1:04 PM by Nick Yeates Average User Rating(0 ratings)

Summary This document is designed to be the official document of record that describes how to create and publish a ZenPack. It does not provide details about the content of a ZenPack, but is restricted to the mechanics of the development process and code distribution. This guide mainly applies to ZenPacks that will be published to the Zenoss community. The majority of topics covered are not necessary for ZenPacks whose value is limited to the local organization. Quick Commands Use these commands as an intial guide or quick reminder. If you want to understand what you are doing, click on the section titles for details. Errors? Need help? Each # links directly to a detailed section # below. Also, feel free to leave comments if you have an issue. <anything_in_brackets> should be replaced by the proper string needed for your circumstances. screencast.png (watch screencast of these commands) # 1 - Checkout a ZenPack

# A - If existing repo (exists on github.com)... Fork and Clone
fork the ZenPack from github.com; done in the web UI, NOT the below command
git clone git@github.com:<your_github_username>/<zenpack_name>.git
cd <zenpack_name>; pwd
git remote add upstream https://github.com/zenoss/<zenpack_name>.git

# B - If New Repo...  Initialize New Repo
mkdir <new_zenpack_name>; cd <new_zenpack_name>
git init

# 2 - Setup / Develop ZenPack

wget https://raw.github.com/zenoss/Community-ZenPacks-SubModules/master/.gitignore

# A - If existing repo (exists on github.com)... Link-Install
zenpack --link --install <zenpack_name>

# B - If New Repo... Relocate Code
Create a ZenPack <shell> (see details below for help)
cp –R $ZENHOME/ZenPacks/<shell_zenPack_name>/ <parent_repo_dir>/
cd <parent_repo_dir>
zenpack --link --install <shell_zenpack_name>; zopectl restart

# 3 - Commit changes locally

git add -A
git status
git commit -m '<Arbitrary commit message>'
git tag -a <NameAndVersionNoSpaces> -m '<Pretty Name And Version>'

# 4 - Install and test

Export ZenPack to .egg
Remove ZP, Install .egg, test features, test uninstall
Final Commit and Tag

# 5 - Pull updates from main zenoss github

git fetch upstream
git merge upstream/master

# 6 - Push changes to personal github

Assure a repo exists on your personal github; done in the web UI
git push --tags origin master

# 7 - Publish to main github repo

Send github "pull request", if existing repo
Fill out the Community ZenPack Submission Form

Table Of Contents Summary Quick Commands Table Of Contents ZenPack Development Overview Detailed Documentation 0) Required: Github Account and Git install 1) Checking Out a ZenPack A) If existing ZenPack (on github.com)... B) If this is a brand new ZenPack... 2) Make Changes / Develop ZenPack A) If existing ZenPack (from github.com)... B) If this is brand new ZenPack... 3) Committing Changes Locally 4) Install and Test ZenPack in Zenoss 5) Pulling latest updates from Main Zenoss Github Repo 6) Pushing Changes to Personal Github 7) Publishing to Main Github Repo A) If existing ZenPack (was already on github.com)... B) If this is a brand new ZenPack... References ZenPack Development Overview Zenoss development is moving completely to Git and is being hosted on Github. The Community ZenPacks are available from the official Zenoss GitHub repository. Individual ZenPacks are located in the root of the zenoss github space at http://github.com/zenoss/. Each Community ZenPack is now its own separate repository. The definitive listing of all community ZenPacks is contained in the SubModule Repository. In the past, the ZenPack organization was setup so that all Community Zenpacks were in one giant repository. Note that this changes much of the former instructions that are out there, so be careful. Subversion access will no longer be available in the future. Repository Diagram

 Distributed_repository_structure_on_github.png

The figure above illustrates the multiple levels of repositories used in ZenPack development. We have defined their names as used throughout this document, and example URLs to where some are located. The ‘main public repositories’ that each reside in various users accounts are read-write to the owner, and read-only to outsiders. Code at these locations are considered the master repository and act as the source for updates. After creating a private account on GitHub, a fork or copy of any repository can be created in ones personal public account. Since ‘personal public forks’ have read/write access, the repository can be modified. Finally, the ‘private cloned repository’ on your local machine or development server is where you do your development actions. It will feed its changes up the chain. Additionally of note is the ‘SubModule Repository’ at the top. This is a zenoss-controlled repository (just one repo), that contains links to all community ZenPacks. It is read-only. Because community ZenPacks are distributed in many places (each of them is located at a different users github space), we use this to keep track of the locations of all ZenPacks. Assure that your ZenPack is in this listing, otherwise it will not be included in official listings and automated testing. Basic workflow for publishing a ZenPack Clone or create Zenpack and git repo Develop and test on local repository (private cloned repository) Push local repository to personal github repository (personal public fork) Fill in Submission Form and/or Send pull request to get content on main public repository Detailed Documentation 0) REQUIRED: GITHUB ACCOUNT AND GIT INSTALL Git and Github are now required for anyone wanting to edit or submit ZenPacks. You will need a github account to share any changes or new ZenPacks back to the community. Make sure to run all instructions at Github’s Setting up Github help page. Despite this requirement, you will mostly use local git commands. Git must be installed locally on your machine and is seperate from github. Additionally, all of our instructions will be based off of you having your own github account and git installation. Checklist: Create a github account - REQUIRED, if you want to share your code Install git and setup Github - IMPORTANT, you must complete the instructions, including SSH-key setup, or nothing below will work Other optional settings 1) CHECKING OUT A ZENPACK A) IF EXISTING ZENPACK (ON GITHUB.COM)... If this ZenPack already exists on http://github.com/zenoss or http://github.com/<some_user>, you will want to fork it. Make a fork on Github.com This is done on the github.com website, while viewing one of the ZenPack repos. Click the “Fork” button when viewing a ZenPack.

 Screen shot 2011-10-03 at 6.15 AMEDT.png

What you are doing is forking (cloning / copying) the existing code (main public repository) from github.com/zenoss/, to a version under your personal github account (personal public fork). See diagram above.

Clone from your fork Make a copy of the forked repository, to your local machine:

You can be in any directory on your machine. The default on linux machines is often /usr/local/git/ . You do not need to create a directory with the name of the ZenPack. $ git clone git@github.com:nyeates/ZenPacks.community.PredictiveThreshold Define where you will push and pull to You may want to do this because it will allow you to push out updates you have made and pull down updates from the main repo at a later date. $ cd ZenPacks.community.PredictiveThreshold; pwd /usr/local/git/ZenPacks.community.PredictiveThreshold

$ git remote -v origin git@github.com:nyeates/ZenPacks.community.PredictiveThreshold (fetch) origin git@github.com:nyeates/ZenPacks.community.PredictiveThreshold (push)

$ git remote add upstream \ https://github.com/zenoss/ZenPacks.community.PredictiveThreshold.git The “cd”, first gets to the proper directory. The “remote -v” says to show (verbosely) the existing remote definitions for the repository that the cwd contains. The “origin” remote is often already made by default. It refers to where you cloned from. The “remote add upstream” defines a new remote definition that says, “upstream” is now known to mean the repo at the ‘main public repository’. We can use this upstream remote later to pull changes that are upstream, down to our local repo. You can get the url used in the upstream command, by visiting the github ZenPack URL (there are two kinds, HTTP and Git Read-Only; either suffices). FIXME fix links in code above - too many colors and markup make it confusing to users B) IF THIS IS A BRAND NEW ZENPACK... If you are working on an existing hosted ZenPack, ignore this section. If you want to create a new ZenPack from scratch, or you have some code developed and want to get it into a new repo, follow these directions. If you need more details than provided below, see this Progit book chapter. If you have new code or files already existing somewhere, ignore the mkdir. Just assure that you are in the root directory with the files of your project. #Best Practices: Create your repo somewhere NOT in $ZENHOME/ZenPacks . We want to keep the development somewhere else. Maybe make a directory that holds all of your developed ZenPacks together - each subdirectory being a seperate git repo. Later, when we install the ZenPack, we will use the –link option to install it in place and not copy the code over to $ZEHHOME/ZenPacks. See more detail below in section “2) B) 2. Relocate the ZenPack to a local private repository” $ mkdir ZenPacks.community.LeastAvailablePortlet $ cd ZenPacks.community.LeastAvailablePortlet

$ git init “git init” initializes the cwd (current working directory) as a new git repository. You can see a .git directory now if you run a “ls -la”. The .git directory contains all the metadata and history involved with this repository. Do not mess with this directory unless you know what you are doing. 2) MAKE CHANGES / DEVELOP ZENPACK The primary reference for developing ZenPacks is the Zenoss Developer’s Guide. A listing of ZenPack development resources is kept up to date. Jane Curry has also published an excellent supplement to the Developer’s Guide: Creating Zenoss ZenPacks. For the purpose of this document, discussion of ZenPack development will be restricted to creating a new ZenPack shell and getting it to interact with the local Git repository. Add .gitignore file First, You need to add a “.gitignore” file to all repositories. The earlier you add it, the more trouble it will save you later on. $ wget https://raw.github.com/zenoss/Community-ZenPacks-SubModules/master/.gitignore A .gitignore file tells git to ignore particular file types and never to include them in the repo or upstream. Files such as .pyc, .pyo and build directories are examples of cruft that we do not want getting into the ZenPack source code repo. An alternate is to use “curl -O url” A) IF EXISTING ZENPACK (FROM GITHUB.COM)... If the ZenPack already exists at http://github.com/zenoss or http://github.com/<some_user> and you have already forked and cloned it locally, you will want to install the ZenPack. $ zenpack –link –install ZenPacks.community.PredictiveThreshold “–link” does some magic here. Instead of copying the code over to $ZENHOME/ZenPacks, it keeps the code in place in your repo, and installs the ZenPack as a linked ZenPack. Think of this like a symlink. It allows you to keep your code in place. Read the reasoning behind this below in B) #2. B) IF THIS IS BRAND NEW ZENPACK... Create a ZenPack Shell in the Zenoss UI CreateAZenPack.png Navigate to Advanced > Settings > ZenPacks. From the Action menu, select Create a ZenPack… In the Create a new ZenPack dialog, provide a name. See Section 3.2.1 in the Zenoss Developer’s Guide for details regarding ZenPack names. Most use “ZenPacks.community.*” For details on dependancy information, please see our Zenoss Developer’s Guide section on “Specifying Dependancies”. As an example, we will create ZenPacks.community.MyZenPack. Relocate the ZenPack to a local private repository

The Zenoss Developer’s Guide says the following about locating ZenPack source code outside of Zenoss:

For any non-trivial ZenPacks we recommend maintaining the source code somewhere other than $ZENHOME/ZenPacks. There are a couple reasons for this: Performing a zenpack –remove If your ZenPack source is maintained in a version control system it is frequently easier to keep the code within a larger checkout directory elsewhere on the filesystem. Copy the new ZenPack files to your local repository directory: $ cp –R $ZENHOME/ZenPacks/ZenPacks.community.LeastAvailablePortlet/ /usr/local/git/ Replace the zenpack name and repository directory (/usr/local/git/ above refers to a common place where git repos are stored). Decide on the path that will be a parent directory containing all your developed repos. You can have your git repos stored anywhere in your directory structure.

Reinstall the ZenPack at the new location: $ cd /usr/local/git/ $ zenpack –link –install ZenPacks.community.LeastAvailablePortlet

 INFO:zen.ZenPackCMD:Previous ZenPack exists with same name ZenPacks.community.FileManager
INFO:zen.ZenPackCMD:installing zenpack ZenPacks.community.FileManager; launching process

$ zopectl restart

“–link” does some magic here. Instead of copying the code over to $ZENHOME/ZenPacks, it keeps the code in place in your repo, and installs the ZenPack as a linked ZenPack. Think of this like a symlink. It allows you to keep your code in place. Read the reasoning behind this above in B) #2.

Check to see that the ZenPack was installed: $ zenpack –list | grep LeastAvail Develop the ZenPack At this point, the ZenPack can be developed to completion, along with intermittent committing in the next step below. Use the development documents noted at the beginning of this section for more information on development guidelines and methods. #Create README.rst Documentation File Individual ZenPack documentation includes a short description of the ZenPack, screenshots of it in action, components of its install, and install / usage instructions. README.markdown.png We host all ZenPack documentation on Github, inside of the code for each ZenPack. The following points explain this in more detail: Documentation exists IN the ZenPack code repository; Create the file while you code README.rst file fully contains the documentation in the ZenPack repositories root directory Will be accessible through public github repository; ex: https://github.com/dpetzel/ZenPacks.community.f5/ Screenshots and images can be added, as seen in: https://raw.github.com/zenoss/ZenPacks.zenoss.OpenStackSwift/master/README.rst Format of text is in “reStructuredText” This means you write it in normal ASCII characters, and they are converted to a pretty html’ized format when viewed Those third parties wanting to edit or add to documentation should fork and then clone the entire ZenPack (see instructions in this doc), edit README.rst, add/commit, push to your own github space, and do a pull request. All of this is covered in this doc and the quick commands at the top of this guide. Dont embarass yourself Create this doc - at the very least, a one sentance description This documentation is what shows BY DEFAULT to anyone viewing the initial page of your ZenPacks github page. Github knows to look for and show any content in README.* View any of the links above and see that the documentation is right there on the front page! Reference examples: https://raw.github.com/zenoss/ZenPacks.zenoss.OpenStackSwift/master/README.rst https://github.com/dpetzel/ZenPacks.community.Varnish3/blob/master/README.rst https://raw.github.com/dpetzel/ZenPacks.community.f5/master/README.rst https://raw.github.com/zenoss/ZenPackTemplate/master/README.rst In the past, ZenPack documentation or what were sometimes called “jump pages” were made on the community portal website (Jive; ex: Advanced Device Details). These are hoped to still exist, as they provide commenting - but that the content will be automatically pulled from the README.rst’s. This is not yet in place - so both README.rst’s and “jump pages” still exist. Also, we formerly used markdown format. This is now deprecated. FIXME Info needs to be pulled from phonegi doc 3) COMMITTING CHANGES LOCALLY The next two steps 3) and 4) are iterative. You will do them over and over as you develop. Commit, test, commit, test. After having made changes to files, you will want to commit your changes. This is like saving a snapshot of your project. Git monitors files in the background and knows what has changed. You still need to specify what will be part of the repository and when to sync it. If you need more details than provided below, see this Progit book chapter. Best Practice: You want to commit often enough so that you can revert back to former states, or see a breif history of your progress in your commits. One commit at the end of a project, is not good practice. Stage and commit files for each session you sit down and code something, or when you get to a functional completion point. Also, use explanative commit messages that you and others can refer to later, to understand the progression of the project. Stage the files You will need to tell git which files you want to commit. Staging files gets them ready and allows you to see what will be committed, before actually committing. $ cd /usr/local/git/ZenPacks.community.LeastAvailablePortlet $ git add -A The above will stage (prepare) all files in the current and child directories to be ready for commit. It does not actually change any files or make any commit action. It just stages the changes, so that you can know for sure what you are committing. You could also run “git add README” or “git add \*.log” or “git rm error.log” to do some variety of files. You will need to run this on all changed files everytime you want to commit, even if you have added the same files in the past. There is a shortcut to avoid using git add altogether (git commit -a -m ‘blah’), but know what you are doing before using it.

Check Staged files (optional) $ git status $ git diff $ git diff –staged “git status” shows what is staged by the prior command. This helps you check your work to make sure that you have staged everything that you want. “git diff” shows the difference, at a character level inside files, between what was formerly commited and what is not yet staged. This shows nothing at this point, because we have already staged above. It might be a helpful command before running git add. “git diff –staged” is more useful here, in that it shows the difference between what was formerly commited and what is staged for commit.

Commit to local repo $ git commit -m ‘Fixed input to allow for UTF-8 encoded characters’ $ git log This snapshots the changes shown in git status to the current branch ( FIXME - not sure if this works same way in branches). The -m is a required comment on what the commit did. Make this short and descriptive to what the change implements or does functionally. Kind of like the comments you put on wikipedia changes.

The command “git log” shows all prior commits. Use this to verify that your commit went through.

Important: Tag if this is a release $ git tag -a PredictiveThreshold1.2.2 -m ‘Predictive Threshold 1.2.2’ This command creates a tag of the current snapshot. This will show up on the gihub.com GUI and allow users to select a version of the code to view. It is very important to do this, so that future automation works and working with various ZenPack versions is easy!

Best Practice: tag vs commit The difference between the -a and -m on the tag command and the -m on commits, is that tags are to be used primarily for denoting release #s. Make a tag only when you want a formal release with a version #. Commits and their messages (-m) are used to mark incremental save points and explain what it is you changed. Make commits often, for example, when you have completed a small fix or added functionality.

4) INSTALL AND TEST ZENPACK IN ZENOSS At this point, you have developed your ZenPack, and committed changes locally. Before you push these changes up the chain so that others can use it, you will want to test your ZenPack. Create the ZenPack egg file Fill in and save all Metadata and Dependency information, then export the ZenPack by selecting the Action button, then Export ZenPack… ExportZenPack.png Remove and install egg version After creating the egg, remove the ZenPack and install it from the egg. I prefer to do this at the command prompt so I can see if any errors occur: $ zenpack –remove ZenPacks.community.PredictiveThreshold $ zenpack –list | grep LeastAvail $ zenpack –install $ZENHOME/export/ZenPacks.community.PredictiveThreshold-1.0-py2.6.egg $ zopectl restart $ zenpack –list | grep LeastAvail “zenpack -list | grep ...” checks zenoss to see if it thinks that the ZenPacks is installed. It is a good way to verify that your commands have succeeded.

Verify that components and features work Walk though each piece of the ZenPack – collectors, daemons, etc… to verify that everything works as it should. If there are problems, remove the ZenPack and reinstall the one in the repository. Fix any problems. Repeat until everything is working.

Take screenshots of the added functionality so they can be incorporated into the document that will describe the ZenPack (the ZenPack jump page).

Test that egg uninstall is clean Remove the ZenPack from the command line: $ zenpack –remove ZenPacks.community.PredictiveThreshold Log back into the Zenoss system and verify that there are no artifacts or problems after removing the ZenPack.

Final Commit and Tag Run one final ‘git commit’ and tag the release with ‘git tag’. See the instructions for this above in section “3) Committing Changes Locally” 5) PULLING LATEST UPDATES FROM MAIN ZENOSS GITHUB REPO Just in case you think that someone may have updated code upstream from your local repo, you can run these commands to get the latest upstream code on your local repo. FIXME needs additional testing Pull down the latest $ git fetch upstream Merge with fetched upstream $ git merge upstream/master 6) PUSHING CHANGES TO PERSONAL GITHUB Lets get it up on http://www.github.com/nyeates/ZenPacks.community.PredictiveThreshold/ ! You have now developed your code, commited a number times, and tested it (or not). This is not to say the project is complete. You are ready to send it out to your ‘personal public fork’ and/or ‘main public repository’. Often, your ‘personal public fork’ is the same thing as the ‘main public repository’ - it depends on who the original author of the ZenPack was. If you are the original author, this step will put your ZenPack to its final location! Best Practice: You should push to your personal github now and then, and not only at functional completeness. All code, working or not, is good to have out there. If it still crashes and you dont want to release it, but you want the code available, develop in an unmerged branch and push this into your personal public fork. The master branch will show prior work, while the branch will show beta work in progress. Assure that the ZenPack Repo exists on Github This may already exist. If you have forked or cloned your repository, this can likely be skipped, as the matching repo already exists on your personal public github. If you are starting a new ZenPack, that does not yet exist outside of your localhost repository, you will need to do this.

While in your Github Dashboard, click on the “New Repository” button. Make the project name, the same as the directory / repo you created.

  New Repository.png

Alternatively, you can do this via the command line with: $ curl -i -u “nyeates:mypassword” -d ‘{”name”: “ZenPacks.community.LeastAvailablePortlet”, “description”: “Creates a portlet that tells you the least available devices”}’ https://api.github.com/user/repos Check what remotes are defined Again, you may already have the right settings if you have forked or cloned your repository. Remotes are needed to help define where alternate repositories live and how to communicate (push, pull) with them. First, check your existing remotes:

$ cd /usr/local/git/ZenPacks.community.LeastAvailablePortlet $ git remote -v origin git@github.com:nyeates/ZenPacks.community.PredictiveThreshold.git (fetch) origin git@github.com:nyeates/ZenPacks.community.PredictiveThreshold.git (push) upstream https://github.com/zenoss/ZenPacks.community.PredictiveThreshold.git (fetch) upstream https://github.com/zenoss/ZenPacks.community.PredictiveThreshold.git (push) “origin” is a special remote that refers to where the repo came from (if it was forked or cloned from somewhere). “upstream” is the remote that refers to where the ‘main public repository’ is located. In the case above, the repository was cloned from my ‘personal public fork’ of the Predictive Threshold ZenPack and it has an upstream remote that was defined in a step from the section before (Checking Out an Existing ZenPack). In this case, I do not need to change anything and you can skip the step below.

Add remote definition, if needed If you have created the ZenPack locally first, you will need to add an “origin” remote. Even though the true origin is from your localhost, the ‘main public repository’ repository is going to become your origin. While in the local repo directory, run: $ git remote add origin git@github.com:nyeates/ZenPacks.community.LeastAvailablePortlet.git The “git@github.com:username/repo.git” syntax is a read+write SSH method. You can only push (write) to this address if you have setup SSH keys with github on the computer that you are using.

Note: An upstream remote is not needed if there is not yet a ‘main public repository’ that exists for it.

Push to remote $ git push –tags origin master This command pushs to the “origin” remote (’personal public fork’ or ‘main public repository’), from the “master” branch (localhost). This is why we defined the “origin” remote earlier, in order to use here. –tags explicitly says to include any and all defined tags. If we didnt do this, new tags would not be pushed. 7) PUBLISHING TO MAIN GITHUB REPO Verify that you have your ZenPack published to your ‘personal public fork’ or ‘main public repository’ (these are sometimes synonymous, sometimes separate - depends who the author is). If it is on github somewhere, we can proceed. There are two scenarios and each is different: A) IF EXISTING ZENPACK (WAS ALREADY ON GITHUB.COM)... If the ZenPack already existed as a ‘Main Public Repository’ (either at http://github.com/zenoss or http://github.com/<some_user> ), send a pull request on Github.com’s webui. Log in to your private GitHub account and view your Dashboard. On the right-hand side of the screen, under Your Repositories, select the repository you have worked on - for example nyeates/ZenPacks.community.LeastAvailablePortlet : githubYourReos.png Once inside the repository, click the ‘Pull Request’ button:

  Screen shot 2011-10-04 at 6.27 PMEDT.png

REQUIRED: Fill out the web form located at: Community ZenPack Submission Form If there is no Pull Request button, it could be because you are the original author of this ZenPack and do not need to send it up to anywhere else. Just fillout the form below.

Note: If you see the below screen, something has gone wrong. Contact us at community@zenoss.com with details. pullreqforked.png B) IF THIS IS A BRAND NEW ZENPACK... The ZenPack does not already exist at http://github.com/zenoss or http://github.com/<some_user> as a ‘Main Public Repository’, Fill out the web form located at: Community ZenPack Submission Form REFERENCES FIXME Fill with links to other content, get some from phonegi doc FIXME Contribution Agreement - mention not needed for public github submissions FIXME Dev Guide - parse through to make sure nothing was missed FIXME Add ZenPack Listing editing, somewhere into doc FIXME Replace /usr/local/git/ with $ZENHOME/Repos or some other thought out directory specfically for repositories Like (0) 16530 Views Tags: zenoss, community, development, zenpack, subversion, zenpacks, developers, repository, git, github Comments (19) daveh daveh Nov 23, 2011 3:33 PM

Note: This comment and the following 3 comments are refering to a much older version of the above document. The old document was drastically different and breif. While it’s good to have a concise set of instructions, this page really needs to be expanded. For some of these steps it is not clear in which direction the updates are occuring. It would be better to stay away from terms like ‘sync with’ in favor of ‘sync from/to’. It would also be good to understand which steps are benign and which could clobber your project. And give a few checkpoints along the way to confirm the steps performed so far are correct. While these steps are helpful in the case of editing files directly from the current shell session, it’s not clear how changes made from other sources are merged in. If I use an external editor/IDE and then copy in new files, I assume they need to be manually added and committed. But what about changes made from Zenoss? If I export the zenpack for example, and zenoss updates my objects.xml, does that need to be manually committed? If so, to which branch: my master or my development branch? Perhaps it is just my inexperience with git but this seems like a lot of trouble for developing just a zenpack. If I were working collaboratively or making changes to other people’s zenpacks (which I don’t even think is possible here) I can see why this would be necessary. But for a single developer working on a single zenpack: is it really necessary to push, pull and merge the entire Zenpack tree? Like (0) Nick Yeates Oct 19, 2011 2:15 AM (in response to daveh)

Dave, thanks for the feedback. These instructions were made a while back, when we were first moving to Github. They are in need of updating. We have an active plan and project to improve Community ZenPack SCM practices. Work is planned for A) the layout of zenpacks on github B) clarification of our suggested dev workflow and C) our dev and github documentation. Simon Jakesch and I are heading this up and we will plan to evaluate and incorporate where-possible, your suggestions. A) Layout We are going to change the structure of our ZenPacks on github. Likely, it will look like this: Each ZenPack will have its own seperate repo. No more cloning the entire tree of ZenPacks. Additionally, we may be moving Zenoss Core development over to github too. B) Workflow Not sure of all what is in store here yet. First, read this superb community-written document in detail: http://community.zenoss.org/docs/DOC-10223 (download the PDF). It is a must-read. It documents the mechanism / workflow for developing Zenoss ZenPacks. It may answer many of your questions. C) Documentation Here are some more of what I want to look into for docs around this: An official and reviewed document that goes over the mechanics of the ZenPack development process. Right now, it is this: http://community.zenoss.org/docs/DOC-10223 I think that there should be a concise / cheat-sheet document, accompanied by additional documents that dive into more detail. I am also thinking that there should be different tutorial use-cases such as: I am totally new to developing with Zenoss, how can I get started I already dev’ed my ZP, how do I put it on github? I want to make some changes on someone else’s public ZenPack. I know how to dev ZPs, but Ive never done git / github. I am still dev’ing a ZP, what do I do? I have developed a ZP entirely in the Zenoss GUI, how do I setup and submit this to github? Git / Github / Zenoss suggested workflow and environment Answers to your questions: Many of your questions surround around the development mechanisms explained in http://community.zenoss.org/docs/DOC-10223 If I use an external editor/IDE and then copy in new files, I assume they need to be manually added and committed. I am thinking that you may have not implemented step “3.3.Relocate the ZenPack to the local repository” of the above doc. Once you have, any edits with an IDE are also now to the git repository. Either as soon as you add any files, or once you get to a commit point (dont wait too long between commits), follow the directions below. How do I add and commit files locally? A: run “git add *”, then run “git status” and finally “git commit -m ‘new code that will introduce select-all functionality’” this assumes that you are in the local repo directory “git add *” will stage (prepare) all files in the current and child directories to be ready for commit. It does not actually change any files or make any commit action. It just stages the changes, so that you can know for sure what you are committing. You could also run “git add README” or “git add \*.log” to do some variety of files. You will need to run this on all changed files everytime you want to commit, even if you have added the same files in the past. There is a shortcut to avoid using git add altogether (git commit -a -m ‘blah’), but know what you are doing before using it. “git status” shows what is staged by the prior command. Helps you check your work to make sure that you have staged everything that you want. “git commit -m” snapshots the changes shown in git status to the current branch(?). The -m is a required comment on what the commit did. Make this short and descriptive to what the change implements or does functionally. Kind of like the comments you put on wikipedia changes. If I export a zenpack and files change, what do I do? A: Again, if you have implemented step “3.3.Relocate the ZenPack to the local repository” of the above doc, there should be no problems here because any changes that the export makes, is a change to the files in the local repository. You could run “git add /ZenPacks/community/IBMSystemxIMM/objects/objects.xml” (”git add *” is easier) and then “git commit” To which branch? I am not so clear yet on branch use-case. I would venture to think that you are making these all on a seperate development branch. You do everything on a dev branch, until you have tested, installed, uninstalled, and know that your work is ready for prime-time. This is, at least, what is depicted by a few of our docs. General Comments on using Git It may seem a little difficult to use github for one-man ZenPack work, but our hope is that more collaboration takes place, that we have one flexible platform for all our code, that others can easily fork and allow access to their code (even without going through us), and to automate our workflow when getting new submissions (no more backlog). Additionally, I look at it as a personal investment. It will be good to be a master at git, so you can use it elsewhere. Like (0) Nick Yeates Oct 19, 2011 2:12 AM

Answers to other common questions: How can I pull down changes made to the main github repo (github.com/zenoss/Community-ZenPacks) ? / How can I sync from the official ZenPack repository? A: all you need to run is: “git fetch upstream” and “git merge upstream/master” “fetch” gets the data from upstream (github) and saves it in a temporary place locally - it is not merged in yet. “merge” actually brings this new data that you got from fetch and merges it with the main local branch (master). Now you have the latest stuff from main github. This assumes that the definition for your upstream remote (github) is in place (check by running “git remote -v”. If its not defined, or you are not sure, use: “git remote add upstream git:github.com/zenoss/Community-Zenpacks.git” ) How can I push my local changes to my own github repo (ex: nyeates/Community-ZenPacks)? A: You first had to have created a github repo (done in the github web gui while logged in, button called ‘New repository’). You ussually name it the same as the repo that you already have locally (ex: testSplit). Next, you define this new remote repo source. While in the local repo dir, run “git remote add origin git@github.com:nyeates/testSplit.git”. Replace with your username and repo name. Finally, push your local repo to the github remote repo with “git push -u origin master”. This pushes to origin (github) from the master branch (local). Like (0) daveh daveh Sep 27, 2011 2:54 PM Thanks Nick, this is quite helpful and answers some of my questions. I did also read David B’s document (mentioned above) and found it very helpful for the github stuff specifically. But it did not have more detail on the actual git operations than what was listed here; that’s why I posted the questions. I should mention that I did relocate the Zenpack to the local repository as describe by David (and also by Jane). I understand why that’s a ‘best practice’ and that part never gave me any problems. I would not have dared sync the Zenpack repository to the Zenoss runtime directory! Most of my problems were clearly git newbie issues. I understand now that my local git repository treats changes the same way regardless of whether I edit a file locally, copy a file in remotely, or allow Zenoss to update a file, as in the case of objects.xml. And these changes require a ‘git add’ and ‘git commit’ prior to merge. Thanks for the tip on ‘git status’ command. Checkpoints are important for us rookies. Without recounting every mistake I made (and there were many :- ) I’ll focus just on this: Merging to github the first time wasn’t so bad, as my Zenpack did not already exist in the tree. But when I updated my Zenpack I was afraid my existing files on github might overwrite my new files when I merged the (github) upstream with my (local) master. And that’s exactly what happened. I won’t go into detail as it was almost certainly ‘pilot error’. But that’s why that section of the instruction should be expanded, and a few checkpoints added. I’m glad to hear you are planning to realign the repositories so individual ZenPack submodules can be developed without having to clone entire ZenPacks tree. That would eliminate the problem I just described. Understood: there are scenarios where the true collaborative environment is required. But for single Zenpack development it would be a welcome simplification. I guess the trick is to align it so you can support both simultaneously: the collaborative environment for advanced users and the individual environment for rookies. Like (0) dpetzel dpetzel Nov 23, 2011 6:06 PM Nick, Does this imply that the links on the zenpack listing page should link to the github repo(markdown page) rather than to the existing jump pages? Rather than maintaining both (when an existing jump page exists) does it make sense to either delete the existing jump page, or remove its contents and replace with a link to the markdown page. The advantage to the second is that comments wouldnt be lost? In the past, ZenPack documentation or what were sometimes called “jump pages” were made on the community portal website (Jive; ex: Advanced Device Details). This is now deprecated. Like (0) Nick Yeates Nov 25, 2011 1:25 PM (in response to dpetzel) I kept the post brief on this to keep it simple. This needs thought and discussion. Basically, it is a large task on our list that needs to be done. It would be great to get community involvement on deciding how to best implement the change and help in implementing the change. For now, maybe keep the jump page with minimal info (initial description and screenshots) (or just keep what is up already) and link off to the github repo for details. As you said, this keeps commenting abilities. What I would love to do is have the jump pages on jive reflect the content in the README.markdown files. Markdown file would become the master document. This would keep commenting that I do not think github has. Actually, we could rid of jump pages if github had comments and a rating system. I did not see such things. Let me know if you see different. Any ideas on how we might ideally set this up and automate it? Thank! -Nick Like (0) dpetzel dpetzel Nov 26, 2011 6:53 PM (in response to Nick Yeates) The jump pages have a few advantages that I can think of: 1) People are used to them. 2) Contents can produce hits when someone searching the community site 3) They allow direct commenting. I think #2 might be the most important. If the pages were moved to github It would now require users to search 3 sites for information (zenoss.org, the ZCA wiki, and then github). Given that a lot of people don’t develop Zenpacks driving those users to Github while searching for answers feels wrong. From my vantage point I think the best compromise is: 1) Making the readme.markdown the “master” document 2) Have some tag or flag in the file that references the jump page. 3) There is an automated routine (or manual script) that would take the markdown, run it through the render, and replace the existing contents of the jump page with the rendered output. Perhaps part of the process that Zenoss performs when accepting the pull request would be to fire off the script to re-populate the jump page contents? off-cuff-example... I add the following two tags to my readme file: <sync_jump_page=True /> <jump_page_url=”http://community.zenoss.org/docs/DOC-11352” /> Upon converting/updating those get stripped from the final output. Meanwhile this gives folks the ability to “subscribe” to the process or not. Like (0) Andrea Consadori Andrea Consadori Dec 15, 2011 12:49 PM Hi Nick, i want to better understand how sync my existing zenpacks so i need few info: 1- i just read all the docs i found but it’s better if i fork my own zenpack or if i create a new version? 2- i try to local copy zenpack but i’ve issues [root@ipmonitoring git]# git clone https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git//andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git Cloning into Community-Zenpacks... Password: error: SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git/info/refs//andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git/info/refs fatal: HTTP request failed [root@ipmonitoring git]# git clone https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git//andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git Cloning into Community-Zenpacks... Password: error: SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git/info/refs//andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git/info/refs fatal: HTTP request failed [root@ipmonitoring git]# git clone andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git fatal: repository ‘andreaconsadori@github.com/andreaconsadori/Community-Zenpacks.git’ does not exist [root@ipmonitoring git]# git clone git@github.com:andreaconsadori/Community-Zenpacks.git Cloning into Community-Zenpacks... The authenticity of host ‘github.com (207.97.227.239)’ can’t be established. RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ‘github.com,207.97.227.239’ (RSA) to the list of known hosts. Permission denied (publickey). fatal: The remote end hung up unexpectedly 3- i’ve some zenpacks that i recover from backup and are not in developer mode, what’s the best way for this zenpack? there’s a tric or i’ve to make local a new zenpack and then copy the template ? regards. p.s. i’m not a coder so i’ve some difficulties with this new github repository methodology. Like (0) Nick Yeates Dec 19, 2011 11:42 AM (in response to Andrea Consadori) Andrea, Sure, I can help here, appologies for the wait. Let me answer your questions one at a time. 1) You asked if you should fork or make new. Let me give two examples. If you have updated the existing ZenPack “ZenPacks.AndreaConsadori.AlliedTelsys” which already exists at https://github.com/zenoss/ZenPacks.AndreaConsadori.AlliedTelsys , then you want to fork then clone this one. If you have a totally new ZenPack that no one has seen before and it is named “ZenPacks.community.CoolNewZenPack”, then you want to create your own. Like (0) Nick Yeates Dec 19, 2011 11:50 AM (in response to Andrea Consadori) 2) The errors that you are getting looks like your SSH key is not setup with github correctly. Both your computer and github.com website need to know the same SSH credentials to agree upon. Please assure that you have followed these instructions. If you have issues even after assuring the above instructions, I found that sometimes my ssh identity became non-active. I fixed this with the following commands “ssh-add -l” to see which ssh identities are active. If none, you can add with “ssh-add ~/.ssh/id_rsa” (or whatever the path to your private RSA identity file). I see that you are trying to clone “github.com/Community-ZenPacks” repository. This is a deprecated repository. We no longer use it. Do not clone this repository. Also, please read through the commands on this page above. They are very different now. Instead, each of your ZenPacks are in individual repositories on their own. Browse them at http://github.com/zenoss or in the Submodule repository. You will first need to fork the repository by clicking “Fork” in the GUI at https://github.com/zenoss/ZenPacks.AndreaConsadori.AlliedTelsys (assure that you are logged in). After it is forked, it now exists at https://github.com/andreaconsadori/ZenPacks.AndreaConsadori.AlliedTelsys . You can now clone it to your local machine with: “git clone git@github.com:andreaconsadori/ZenPacks.AndreaConsadori.AlliedTelsys.git” Like (0) Nick Yeates Dec 19, 2011 12:15 PM (in response to Andrea Consadori) 3) There is a trick to getting ZenPacks back to Developer mode. See Restoring ZenPacks to Development Mode . You are lilkely to use the second option. Like (0) jcurry jcurry Dec 21, 2011 9:20 AM We suddenly seem to have moved from markdown as a documentation standard to README.rst (having just worked out markdown). Is there a recommended Zenoss template for README.rst? Thanks, Jane Like (0) Nick Yeates Dec 21, 2011 1:06 PM (in response to jcurry) This is true Jane - appologies for jumping around. The background, is that markdown was originally chosen, and then later, Chet did research into various solutions and also Egor spoke up for reStructuredText. The research and community voice for rst showed that it would meet our needs better. So far, this has been a silent switch. Should it be announced? If so, where might you suggest? Reference README.rst info can be found here: http://community.zenoss.org/docs/DOC-8495#README.rst Reference examples: https://raw.github.com/zenoss/ZenPacks.zenoss.OpenStackSwift/master/README.rst https://github.com/dpetzel/ZenPacks.community.Varnish3/blob/master/README.rst https://raw.github.com/dpetzel/ZenPacks.community.f5/master/README.rst https://raw.github.com/zenoss/ZenPackTemplate/master/README.rst Like (0) jcurry jcurry Dec 21, 2011 1:17 PM Another question on holding ZenPacks on git. By default, the .gitignore excludes build and dist so what is the recommended way for providing the egg file? Cheers, Jane Like (0) Nick Yeates Dec 21, 2011 4:28 PM (in response to jcurry) Jane, good question - keep them coming. See as an example: https://github.com/dpetzel/ZenPacks.community.f5/downloads Visit one of your own ZenPacks, while logged into github. For example: https://github.com/jcurry/ZenPacks.skills1st.MenuExamples Now click on the “Downloads” tab, or simply tack on “/downloads” to the url to make it: https://github.com/jcurry/ZenPacks.skills1st.MenuExamples/downloads In this area, you can upload your own files - .egg files. And guess what? No more .egg.zip’s! Also given as an option to outside users is the ability to download the code in .zip or .tar.gz formats. Dont confuse this with you still having to upload the compiled .egg’s The .egg that you put in this location can also be referenced in the README.rst as a simple URL. An example of this URL might be: https://github.com/downloads/jcurry/ZenPacks.skills1st.MenuExamples/ZenPacks.skills1st.MenuExamples-1.0-py2.6.egg So, all said and done, README.rst will supplant jive “jump pages”, which will have links to .egg’s. “Jump pages” are still planned to exist, and will be copies of the in-repo README.rst’s . Like (0) jcurry jcurry Dec 22, 2011 7:09 AM (in response to Nick Yeates) Too complex! Too may options! We have to make this as simple as possible if people are going to use it. I have decided to change the .gitignore to include dist and *.egg files. Then, when you add/commit/push it includes the dist directory with the egg file. In README.rst (which definitely does need documenting, publicising and template providing) you can then provide a link to your dist/<zenpack>.egg file. Download ======== Download the appropriate package for your Zenoss version from the list below. * Zenoss 3.0+ `Latest Package for Python 2.6`_ .. External References Below. Nothing Below This Line Should Be Rendered .. _Latest Package for Python 2.6: https://github.com/jcurry/ZenPacks.ZenSystems.olsonPower/blob/master/dist/ZenPacks.ZenSystems.olsonPower-1.2-py2.6.egg?raw=true Note that the directory for dist/<zenpack> seems to need /blob/master/ inserted before dist - more git magic I guess. Cheers, Jane Like (0) dpetzel dpetzel Dec 22, 2011 1:02 PM (in response to jcurry) First to Jane’s direct comments, I’d suggest not altering your .gitignore, but rather leverage the downloads section of GitHub. With the approach you’ve taken, Downloading an older version of the EGG becomes more challenging. I understand that you can go backward intime through the commit history if need be to get an older EGG, but every time you push, the only egg that is displayed is the most up to date. Perhaps this is the desired effect, but as a user I like having the option to pick any version I want. So each time a pack is ‘released’ that EGG can be uploaded to the downloads section of your Repo and users can make a choice on which version they want, or they simply click your ‘latest’ link in your file. Again this is my personal opinion and not a “you should do this statement” so take it or leave it. As for the larger discussion, I think this discussion is much larger than this thread, but some thoughts I’ve had going through the process. I don’t believe the process has too many options or is too complex for ALL, but that is 100% a true statement for some. Let me try to explain. Using an SCM system for the first time is complicated at first. I don’t care if its GIT, Mercurial, SVN, <insert SCM name here>. The reality of it is it takes some practical hands on for SCMs to make sense. For some folks that develop on a regular basis this is common day, so its second nature to them. For a large percentage of the user base I have to assume they are NOT developing on a daily basis, and as such source control in general is a foriegn concept. Your average System Administrator probably doesn’t work with source control much (if at all). One of the common ‘gripes’ I’ve read in the IRC channel and the forums is that “Zenoss Developers/Engineers” need to provide more documentation and best practices. I believe this is exactly what this document is trying to provide, and to that effort I think its a valuable document. If your writing code, putting it into source control is always a good idea, and to that end Github is a decent option. It has a lot of features such as SCM, wiki, issue tracking, and build/release downloads. So for providing an accessible system that ‘developers’ can use to collaborate and share code this is great step. The learning curve involved with git if your are already familiar with another SCM is fairly low (IMHO). Now that aside, how many people are “writing code” vs simply creating Monitoring templates using SNMP or perhaps Nagios/Cacti plugins? I don’t know the answer to what the percentage is, but the reality is I think its safe to say more than 0 people are contributing ZenPacks that include no custom code. So for someone that doesn’t write code, but wants to contribute something back in the form of a monitoring template created in the UI this process is waaaayy overkill. I think there needs to be some middle ground here. If I’m an SA that hacked up a quick template for an application I support, I should be able to contribute that back without the need to create a GitHub account, learn how to use GIT, learn reStructuredText, etc. I want to re-iterate I think storing the ZenPack source code on GitHub is a good idea, and I can also appreciate wanting ALL ZenPacks in one place. However, I think there needs to be an avenue for the more casual contributor. Whether that be the ZCA or actual Zenoss Inc resources, someone should be able to hand off their exported egg. Like (0) jcurry jcurry Jan 4, 2012 8:25 AM (in response to Nick Yeates) Why are there two different URLs to a downloads area and which should we use??? You quote https://github.com/downloads/jcurry/ZenPacks.skills1st.MenuExamples/ZenPacks.skills1st.MenuExamples-1.0-py2.6.egg which does work provided you type it all in - note the dowloads before the name in the URL - but this doesn’t satisfy the sensible comment from dpetzel about having access to a variety of downloads. To get there, you need a URL like https://github.com/jcurry/ZenPacks.skills1st.MenuExamples/downloads where presumably you can have multiple files for download. Cheers, Jane Like (0) Robert Martin Robert Martin Jun 30, 2012 12:34 PM Would it make sense to have an extra directory layer so that the material, such as source code, could be held in a subdirectory within the same repository as the Zenpack, but the Zenpack itself is a separate subdirectory? To me it seems odd that specific source code, as for example for a Flash plugin, is held in a completely different repository.