Thursday, June 04, 2015

mu-repo 1.1.1 (cloning multiple git repositories)

One tool I use daily for my git workflows (which are almost always composed of several git repositories) is mu-repo (which is a tool written in Python for dealing with multiple git repositories), so, I'm happy to announce that version 1.1.1 has just been released with a workflow which allows cloning multiple git repositories.

See: http://fabioz.github.io/mu-repo/cloning/ for details.

Also, given that it's gaining some traction recently, I decided to spend some time to actually do a homepage for it to help newcomers (the README on Github was becoming too big already):

http://fabioz.github.io/mu-repo/

As a note, I believe even long time users may benefit from taking a quick look at the Tips & Tricks section ;)

p.s.: mu-repo can now be installed through pip: pip install mu-repo.

p.s.2: Kudos to http://www.mkdocs.org for providing a nice tool to build a homepage from Markdown docs :)

Wednesday, May 27, 2015

PyDev 4.1.0 (code completion improvements, Python 3.5 grammar)

Ok, PyDev 4.1.0 is out now.

Improvements in this release include supporting more use cases when unpacking lists/tuples/sets -- this is the current focus in PyDev right now: making its type inference/code-completion best in class ;)

Also, the new constructs on Python 3.5 are already supported in PyDev in its Python 3 grammar (i.e.: async/await, and @ matrix multiplication).

Another nice feature added in this release is that it's possible to select constructs to be folded by default when an editor is opened (as the image below shows):




Thursday, April 23, 2015

Wrapping docstrings/comments in PyDev

This is a nifty trick when working with PyDev...

When you're writing docstrings it's usually pretty annoying having to manually wrap the contents of the string to fix inside a proper number of columns: say your standard is 80 columns and you've just written 5 lines of comments to fit in that, then, you decide to edit and add something in the middle: there it goes: you have to reindent all the lines below so that the block is nice again.

Well, PyDev has an utility for that: put your cursor on that block and do Ctrl+2, W (i.e.: press Ctrl+2 and type the char W later on). Doing that will fix your comment block to fit in the number of columns available (note that it works for code in docstrings and comments alike).

To give an example, say you have the code below:


Just place the cursor in the block, use Ctrl+2, W and voila, you get:


Hope PyDev users enjoy the trick (aligning those manually is definitely on the annoying group).

As a note, to configure the number of columns to be wrapped to go to preferences > general > editors > text editors and change the 'print margin column'.

To finish, Ctrl+2 has other interesting things (if you just press Ctrl+2 and wait a bit, a popup with the options will appear).

Tuesday, April 21, 2015

Type hinting on Python

If you missed it, it seems right now there's a long thread going on related to the type-hinting PEP:

https://mail.python.org/pipermail/python-dev/2015-April/139221.html

It seems there's a lot of debate and I think the outcome will shape how Python code will look some years from now, so, I thought I'd give one more opinion to juice things up :)

The main thing going for the proposal is getting errors earlier by doing type-checking (which I find a bit odd in the Python world with duck-typing, when many times a docstring could say it's expecting a list but I could pass a list-like object and could get fine with it).

The main point raised against it is is that with the current proposal, the code becomes harder to read.

Example:

def zipmap(f: Callable[[int, int], int], xx: List[int], yy: List[int]) -> List[Tuple[int, int, int]]:

Sure, IDEs -- such as PyDev :) --  will probably have to improve their syntax highlighting so that you could distinguish things better, but I agree that this format is considerably less readable. Note that Python 3 already has the syntax in-place but currently it seems it's very seldomly used -- http://mypy-lang.org/ seems to be the exception :)

Now, for me personally, the current status quo in this regard is reasonable: Python doesn't go into java nor c++ land and keeps working with duck-typing, and Python code can still be documented so that clients knows what kinds of objects are expected as parameters.

I.e.: the code above would be:

def zipmap(f, xx, yy):
    '''
    :type f: callable(tuple(int, int))->int
    :type xx: list(int)
    :type yy: list(int)
    :rtype: list(tuple(int, int, int))
    '''

Many IDEs can already understand that and give you proper code-completion from that -- at least I know PyDev does :)

The only downside which I see in the current status quo is that the format of the docstrings isn't standardized and there's no static check for it (if you want to opt-in the type-checking world), but both are fixable: standardize it (there could be a grammar for docstrings which define types) and have a tool which parses the docstrings and does runtime checks (using the profiler hook for that shouldn't be that hard... and the overhead should be within reasonable limits -- if you're doing type checking with types from annotations, it should have a comparable overhead anyways).

Personally, I think that this approach has the same benefits without the argument against it (which is a harder to read syntax)...

If more people share this view of the Python world, I may even try to write a runtime type-checking tool based on docstrings in the future :)

Wednesday, April 15, 2015

PyDev 4.0 released. Yay!

PyDev just reached version 4.0 --- after 11+ years that's not so much, is it? ;)

The major improvement in this release is that PyDev can now unpack compound types when doing code-completion (i.e.: list(str), dict(int:str), etc).

The screenshot below shows a completion unpacking types by analyzing the current scope:



The second example shows a completion unpacking types by analyzing a docstring (see: http://pydev.org/manual_adv_type_hints.html for more details on the formats supported).



This release also enables Unicode literals and Byte literals to have different colors, so, it's easy to spot what you're working with (and it should respect the semantics of Python 2 from __future__ import unicode_literals and Python 3).

Another nice feature for those that work more in the interactive side of Python is that PyDev now allows users to define a template to send a custom command to the interactive console.

PyDev already has an F2 binding which sends a line to the console, but let's say that you do something else a lot in the console (such as plot(variable)). With this feature you can bind something as Ctrl+F2 to send the selected text as plot(${text}) to the console (the screenshot below shows how that configuration would be achieved).



Also, the PyDev update site is now hosted on http://bintray.com -- note that the update site urls didn't change, so, this change should (hopefully) be imperceptible to end-users...

There are also other nice things (see http://pydev.org for more details).

To finish, a special thanks to all the PyDev supporters (it definitely wouldn't be possible without your help -- for those that want to help supporting it, please access the related links at http://pydev.org).




Friday, March 06, 2015

Navigating through your code when in PyDev

One of the main advantages of using an IDE is being able to navigate through your code in a really fast way, so, I'll explain below how to find what you want when coding in PyDev.

1. Any file in your workspace


The simplest way is searching for any resource for any project you have available (this is actually provided by Eclipse itself). So, if you know the file name (or at least part of it), use Ctrl+Shift+R and filter using the Open Resource dialog:


2. Any Python Token


Another way of getting to what you want is using the PyDev Globals Browser (using Ctrl+Shift+T).

This is actually what I use most as a single place can be used to filter for package names, classes, methods and attributes inside your own projects or any other package in Python itself. Also, it allows for advanced filtering, so, you can search for tokens only inside a package (i.e.: dj.tz filters for any 'tz' token inside django in the example below).


3. Quick Outline (current editor)


Ctrl+O will show a Quick Outline which shows the structure of your current file. And pressing Ctrl+O one more time in this dialog will also show the structure of superclasses in the hierarchy (you can see that in the example below __setitem__ appears twice, once for the method in this class and another one for the superclass).


4. Selection History


Go back and forward in your selection: Alt+Left goes to the place you were before and Alt+Right to the place you just went from... this allows you to easily go navigate through your recent places.


5. Open files


To filter through the open files you can use Ctrl+E: a dropdown will appear and from there you can filter through its name and you can close existing editors using Del from that dropdown too.


6. Go to previous next token (class or method)


Ctrl+Shift+Up and Ctrl+Shift+Down allows you to quickly navigate from your current position to the previous or next method (selecting the full method/class name).


7. Navigate through occurrences and errors in the file


Ctrl+Dot allows navigating through occurrences and errors found in the file, so, in the case below we'll navigate through the occurrences of 'func'.



8. Go to some view/menu/action/preference


This is an Eclipse standard mechanism: using Ctrl+3 allows you to navigate to any part of the IDE.


9. References


Ctrl+Shift+G will make a search showing all the references to the token under the cursor (and the search view where results are shown can be navigated with Ctrl+Dot).


10. Go to definition


Just press F3 or Ctrl+Click some available token and go directly to the selected place.

11. Hierarchy View


Using F4 shows a hierarchy view where you can see the structure of your classes.


12. Show In


Alt+Shift+W allows you to see the current file in a given place (such as the PyDev Package Explorer or the System Explorer from your OS) or your current class/method in the Outline View.


Others


The ones below are available in standard Eclipse and you should also definitely know about it :)

Ctrl+L allows you to navigate to a given line in your current editor.
Ctrl+Q goes to the place where the last edition was made.
Ctrl+F6 navigates through the opened editors. In LiClipse Ctrl+Tab is also bound to it by default-- and I suggest you also add this binding if you aren't using LiClipse :)
Ctrl+F7 navigates through opened views (i.e.: Package Explorer, Outline, etc.)
Ctrl+F8 navigates through opened perspectives (i.e.: PyDev perspective, Debug perspective, etc).
Ctrl+F10 opens the menu for the current view (so you can select filters in the Package Explorer, etc.)
F12 focuses the editor (so, you can go from any view to the editor)
Ctrl+H Opens the search dialog so you can do text searches
Ctrl+Shift+L twice goes to the keybindings preferences

Now you can enjoy going really fast to any place you wish inside PyDev!


Thursday, February 26, 2015

Design for client-side applications in Python (but applicable to other languages too)


Ok, so, this is a post I wanted to write for some time already and never really got the time to do...

First off, this is a design I've worked some times during the years on client-side applications (Python and Javascript). It revolves around some simple concepts which can be used to develop small to big sized applications, and is suited to places where there are more long-lived instances -- which is usually the case on client-side applications (and usually not for server-side web applications).

These are also the concepts I used when implementing PyVmMonitor (http://pyvmmonitor.com).

For those interested, although PyVmMonitor is closed source, the non-domain specific bits are actually open source and may be found at the links below (so, they may be cloned in git and I'll reference them in this post):

https://github.com/fabioz/pyvmmonitor-framework
https://github.com/fabioz/pyvmmonitor-core

1. Plugin system based on interfaces (interfaces in this case being the java concept of interfaces or ABCs in Python) -- usually I like to be explicit about the interfaces provided and registering the implementors for those interfaces (if you want your programs to be extensible, you really should be programming based on interfaces -- for me it helps to think how a client would consume it, instead of thinking about issues on how to actually implement it).

This is also the mechanism that provides dependency-injection (so, you can swap out some implementor -- although it's not the classing dependency-injection because you still ask for things instead of them appearing magically for you at some variable).

The structure would be something like:


pm = PluginManager()
pm.register(EPView, 'my.View') # As a note, EP is a short for 'Extension Point'.
pm.register(EPMenu, 'my.Menu1') # Implementors are registered as strings to avoid 
pm.register(EPMenu, 'my.Menu2') # having to import the classes (to cut on startup time).


An example of use would be:


# Keeps the EPView instance alive inside the PluginManager 
# (more details ahead on item #2) and starts the view main loop.
view = pm.get_instance(EPView).main_loop()

# The EPView implementation could create its menus by asking the EPMenus registered.
menus = pm.get_implementations(EPMenu)
for menu in menus:
    view.create_menu(menu)


The actual implementation that PyVmMonitor is using can be found at https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/plugins.py

You can see that we're not worried about discoverability as most plugin frameworks are, as we're being explicit about registering things (and it should be easy to add that to the structure if we do need the extensibility for clients).

Also, in this particular implementation, I actually skipped the plugins as things are self contained and I didn't want the added complexity, but usually you'd register plugins and then the extension points and implementations would be registered only through plugins (and would specify the dependencies to other plugins).

2. A place to hold your instances:

I like to keep track of where instances I created are... usually it's difficult to track things in long-lived applications (which is usually not a problem in web-based applications because the model is really the database and objects are short-lived).

In the Eclipse SDK for instance it's easy to see many singleton-like structures spread apart many places and there's no unified approach to it. Every place has an API, so, there's a platform which has a workbench which has a part which has an editor... (I know, that's plain object orientation, but I find that lacking when extending things and there's always a different API for accessing anything).

So, instead of using many different APIs there are 2 main places where instances live -- and that's also the place to query for instances:

  •  The PluginManager itself: it's the place to ask for any extension/service in the application, so, it's fair that it can also keep the needed references alive (and we could also query for existing services/extensions in a shell/debugger).
  •  Inside the PluginManager, an extension named EPModelsContainer is provided and it's the place to put the actual models (in a tree-like structure) to compose your domain (for instance, in PyVmMonitor a new AttachedProcess is created for each attached process and has as a child a model for the the actual monitoring, where the statistics gathered are kept in the client side).

Note that this means that the 'ownership' of the items is one of these 2 places -- and they're quite different: when it's in the PluginManager, it'll be kept alive until the application finishes (although it's lazily started). For items in the EPModelsContainer, when any instance leaves the EPModelsContainer it should be readily deleted as well as anything it holds (so, other places should only keep a weak-reference or should monitor the removal of the item from the models container to make the proper cleanup so that the object can be garbage collected at that point).

The actual extension point that PyVmMonitor is using can be found at https://github.com/fabioz/pyvmmonitor-framework/blob/master/pyvmmonitor_framework/extensions/ep_models_container.py (note that it provides classes-based filtering and tree-iteration and could be extended to provide a JQuery-like api to query it).

Also, test-cases should test that before/after each test, all the instances created in the PluginManager and in the EPModelsContainer are garbage-collected! Note that on non-deterministic garbage-collection implementations such as PyPy/Jython this is not feasible because they aren't immediately collected, but on CPython with reference counting, this works great. So, at least test on CPython -- then if you don't have binary dependencies, run on PyPy :)

3. Callbacks:

As we've just defined that the instances ownership is really on the EPModelsContainer, there's room for callbacks which only keep weak-references to bound methods when you're interested is something -- in which case https://github.com/fabioz/pyvmmonitor-core/blob/master/pyvmmonitor_core/callback.py is a pretty good implementation for that... note that for top-level methods, strong references are kept.

Why you may ask? Well, the main reason is that this use-case is usually for closures, so, it may be hard to find a place to add the function to -- and if it's a top level, the function will be alive until the end of times anyway (i.e.: process shutdown).

Anyways, this is probably a case that should only be used with care as unregistering must be explicit and things in the function scope will be kept alive!

4. A standard selection mechanism.

Again, as we just defined that all our model lives inside EPModelsContainer, the selection is usually simply keeping the id of the object(s) selected.

In PyVmMonitor, the base extension for this is the EPSelectionService. The concept is pretty simple: clients can listen to changes in the selection (through a Callback), can trigger selection changes and can get the current selection. As PyVmMonitor accepts multiple selection to inspect multiple processes at once, it always deals with a list(obj_id) and clients act accordingly to the selection to show the proper UI.

The extension interface used in PyVmMonitor lives at https://github.com/fabioz/pyvmmonitor-framework/blob/master/pyvmmonitor_framework/extensions/ep_selection_service.py

5. Undo/redo: Well, in PyVmMonitor there's really no undo/redo functionality because it's not really required for what it does, but on other applications I worked that had undo/redo, the basis was actually on the model entities that entered the EPModelsContainer: if they implemented an interface saying that they provided changes in them, they'd be tracked and commands would be automatically added to a command list for undo/redo purposes when the model changed (simply by recording the id of the object and the attributes new/previous values -- as well as providing a memento for the specific object when it entered/left the EPModelsContainer)... This is a bit different from using the command pattern because it's all done from the outside as we'd actually be hearing changes in the model instead of actively changing how we code to add the command pattern (which IMHO makes code much more verbose than it needs to be).

6. The actual UI... Well, for PyVmMonitor I didn't go through the effort of actually creating a reusable UI, and other projects I worked in which did have that concern aren't actually mine to open source, but the idea here would be making a view which would query for EPMenu, EPToolbar, EPAction, EPCentralWidget, EPDock, etc and would create the actual UI from that. For PyVmMonitor I just have a simple non-extensible UI which listens to the EPSelectionService and updates its central editor accordingly, and the UI has a 'def set_model' which receives the actual model or models it should show.

Well, that's it, hope you liked the approach, as I said, I've used it some times in the last years and it works great for me (but I'm interested if there are better approaches out there which could be reused or improve on those aspects).