Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      *               255.255.0.0     U     0      0        0 eth0

To get the output of the command we will use subprocess. So our python code will look like this.

import subprocess

p = subprocess.run("route",stdout = subprocess.PIPE)
lines = p.stdout.decode("utf8").splitlines()
headers = lines[1].split()

But how to create a nice datastructure to hold routes ? The answer is namedtuple.

Route = namedtuple("Route",headers)
routes = [Route(*line.split()) for line in lines[2:]]

route = routes[0]
print(route)
print("as a tuple %s" % route[0])
print("as a class like structure" % route.Destination)
print("as a dictionary %s" % route._asdict())

The output is.

Route(Destination='default', Gateway='172.17.0.1', Genmask='0.0.0.0', Flags='UG', Metric='0', Ref='0', Use='0', Iface='eth0')
as a tuple : default
as a class like structure : default
as a dictionary : OrderedDict([('Destination', 'default'), ('Gateway', '172.17.0.1'), ('Genmask', '0.0.0.0'), ('Flags', 'UG'), ('Metric', '0'), ('Ref', '0'), ('Use', '0'), ('Iface', 'eth0')])

The flexibility of python never ceases to amaze me, and I am always finding hidden gems and quicker ways to do things.

class RpcClient(object): 
    def rpc_list_users(self,*args,**kwArgs):
        result = self.call("rpc_list_users",args,kwArgs)
        return result

    def rpc_get_user(self,*args,**kwArgs):
        result = self.call("rpc_get_user",args,kwArgs)
        return result

The are over 20 functions on the RPC server, I didn’t feel like typing out all of the code for these functions. Also what if I added more rpc functions on the server ?. The solution in this case was metaclasses.

Note this demo uses python 3, the syntax for metaclasses with slightly different.

Understanding python classes and objects

To understand how to use metaclasses, one must understand how python classes and objects work behind the scenes. A great way to explore this is with ipython or ipython notebooks which is part of the juypter project.

Simply put classes and objects are nothing more then dictionaries. You can verify this yourself using your python interpretor.

class Foo(object):
    a_class_variable = "a"
    def a_function(self):
        print(a_function)
        
pp("Foo.__dict__")
pp(Foo.__dict__)
pp("pretty Foo")
pp("Pretty Foo.__dict__")
pp(dict(Foo.__dict__)) #convert it to a dictionary so pp dumps it out nice

Output from running above code.

'Foo.__dict__'
mappingproxy({'__doc__': None, '__weakref__': <attribute '__weakref__' of 'Foo' objects>, 'a_class_variable': 'a', 'a_function': <function Foo.a_function at 0x7fb358f8d6a8>, '__dict__': <attribute '__dict__' of 'Foo' objects>, '__module__': '__main__'})
'pretty Foo'
'Pretty Foo.__dict__'
{'__dict__': <attribute '__dict__' of 'Foo' objects>,
 '__doc__': None,
 '__module__': '__main__',
 '__weakref__': <attribute '__weakref__' of 'Foo' objects>,
 'a_class_variable': 'a',
 'a_function': <function Foo.a_function at 0x7fb358f8d6a8>}

Inside all python classes, there is a variable called __dict__. If we print it out, we see that type mappingproxy which if we look inside the python docs says that it is type of dictionary which can not be modified.

Inside the dict variable we see two values:

• ‘a_class_variable’: ‘a’
• ‘a_function’: <function Foo.a_function at 0x7fb358f8d6a8>

If we could modify Foo.__dict__ we could add methods and variables to class Foo at runtime. So we would want to do something like this.

def my_function(self):
    pass

Foo.__dict__["my_function"] = my_function
Foo.__dict__["my_variable"] = 99 

But the mappingproxy object will not allow itself to be modified. So how can we do this ? With a metaclass.

What is a metaclass

In simple english a metaclass is a class which inherits off class type and allows one to modify the class’s mappingproxy. Here is a simple example.

class BarMetaClass(type):
    def __new__(cls, name,bases, dct):
        #this is called before the class has been created
        pp("__new__ \n cls => %r \n name => %r \n bases => %r \n dct => %r" % (cls,name,bases,dct))
        #the parent __new__ metaclass has to be returned
        return type.__new__(cls,name, bases, dct)
    
    def __init__(cls, name,bases, dct):
        #the parent __init__ metaclass method has to be called
        type.__init__(cls,name, bases, dct)
        #this is called after the class has been created
        pp("__init__ \n cls => %r \n name => %r \n bases => %r \n dct => %r" % (cls,name,bases,dct))
        

class Bar(object,metaclass = BarMetaClass):
    a_class_variable = "a"
    def a_function(self):
        print(a_function)

And here is the output from the code.

('__new__ \n'
 " cls => <class '__main__.BarMetaClass'> \n"
 " name => 'Bar' \n"
 " bases => (<class 'object'>,) \n"
 " dct => {'__qualname__': 'Bar', 'a_class_variable': 'a', '__module__': "
 "'__main__', 'a_function': <function Bar.a_function at 0x7fb358f43598>}")
('__init__ \n'
 " cls => <class '__main__.Bar'> \n"
 " name => 'Bar' \n"
 " bases => (<class 'object'>,) \n"
 " dct => {'__qualname__': 'Bar', 'a_class_variable': 'a', '__module__': "
 "'__main__', 'a_function': <function Bar.a_function at 0x7fb358f43598>}")

Simple Explanation what is going on

The metaclass of a class in python 3 is set like this class Bar(object,metaclass = BarMetaClass). All Metaclasses are inherited off class type. All class type classes contain two constuctors which will be called when the type class is being created. These two constructors are.

• def __new__(cls, name, bases, dct) is called before the class is created. It must return the parent metaclass new method which in this case is type.__new__
• def __init__(cls, name, bases, dct): is called after the class is created. Somewhere in the function it much called the parent’s init function which in this case is type.__init__

Both of these functions container 4 arguments.

• cls : this refers to the metaclass itself. It is equivalent to self in a regular object method. So if the metaclass had other methods they could be called like this cls.somemethod(cls,arg). Remember metaclasses are only called when the class is being created. So in the above example, class Bar can not access the methods of class BarMetaClass.
• name : The name of the class being created. So in the above example the name is “Bar”.
• base : which classes it inherits off, in this case “Bar” inherits off “object”
• dct : this equivalent to Bar.dict except it is a regular python dictionary. This means we can modify it.

Since cls,name,base and dct are normal python structures, that means we can modify them before we call the parent’s new function. Before when we wattempted to modify Foo.dict python would not allow use to do this. Using a metaclass this is possible. It done in the new method of the metaclass. Here is a simple example.

def my_function(self):
    return "from my function"

class FooMetaClass(type):
    def __new__(cls, name,bases, dct):
        dct["my_function"] = my_function
        dct["my_variable"] = "my_variable"
        #the parent __new__ metaclass has to be returned
        return type.__new__(cls,name, bases, dct)        

class Foo(object,metaclass = FooMetaClass):
    a_class_variable = "a"
    def a_function(self):
        print(a_function)
        
pp("testing out foo")
pp("Foo.my_variable => %r" % Foo.my_variable)
foo = Foo()
pp("foo.my_function() => %r" % foo.my_function())content

When we run the above example we get the following output.

'testing out foo'
"Foo.my_variable => 'my_variable'"
"foo.my_function() => 'from my function'"

We can see that we have added a class variable and a class method to foo. Ofcourse in this case we could have done this in the class defintion of Foo, there was no need to use a metaclass, but it is just an example to show how it is done.

Solving our orignal problem

In our orignal problem we needed a client class which we needed to add methods to. Each method was basically the same except for the name of the rpc call. So we need a class or a function which creates a function which is similar to this

def rpc_list_users(self,*args,**kwArgs):
    result = await self.call("rpc_list_users",args,kwArgs)
    return result

We notice that the “rpc_list_users” is the only thing changing. So my solution is the make a class like this.

def CreateRpcFunction(name):
    def wrappedFunction(self,*args,**kwArgs):
        self.call(name,*args,**kwArgs)
        return name
    return wrappedFunction

There are other possible ways to do this. In this case we have created what I believe is known as a closure. We then put our function creater to use like this. wrapFunctionNames contains a list of rpc methods in the serve, our metaclass creates the required functions on the client.

wrapFunctionNames = ["rpc_list_users","rpc_get_user","rpc_set_user"]

class RpcClientMeta(type):
    def __new__(cls,name,bases, dct):
        for name in wrapFunctionNames :
            dct[name] = CreateRpcFunction(name)
        return type.__new__(cls,name,bases, dct)
        

class RpcClient(object,metaclass = RpcClientMeta):
        
    def call(self,name,*args,**kwArgs):
        """
        simulates calling the rpc on the server.
        """
        pp("called name %r args %r kwArgs %r" % (name,args,kwArgs))
        return name
        
rpcClient = RpcClient()
pp("dump rpc functions")
pp([name for name in dir(rpcClient) if name.startswith("rpc_")])

pp("rpcClient.rpc_get_user() => %r" % rpcClient.rpc_get_user())
pp("rpcClient.rpc_list_user() => %r" % rpcClient.rpc_list_users(startsWith = "Fred"))

The output of the above code looks like this.

'dump rpc functions'
['rpc_get_user', 'rpc_list_users', 'rpc_set_user']
"called name 'rpc_get_user' args () kwArgs {}"
"rpcClient.rpc_get_user() => 'rpc_get_user'"
"called name 'rpc_list_users' args () kwArgs {'startsWith': 'Fred'}"
"rpcClient.rpc_list_user() => 'rpc_list_users'"

Summary

This is just a simple example of how to use a metaclass to create class method at runtime. Once you understand the basics of metaclasses, then the more advanced tutorials start to make sense. In the above example we could have queried the server for a list of rpc functions it provides. Then we could have created the appropriate functions on the client using a metaclass. This is the magic that many python database libraries use. The query the database and then generate the classes at runtime using metaclasses.

The best way to learn about metaclasses is to use a good python console such as ipython or jupyter notebooks (formerly ipython notebooks). The advantage is that you can modify things at runtime and experiment with them.

All of the code for this post is in an ipython notebook located here.

Sample Heading

Sample Heading 2

Jekyll also offers powerful support for code snippets:

def print_hi(name)
  puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.

Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo.

Once each VPS was up and running, controling each VPS was a breeze with rpyc, it was how to run shell commands on each VPS to set things up. In the rpyc documentation it mentions another module called plumbum which uses a python syntax to do shell commands. More importantly it will ssh into another machine and run them there. You can do ssh commands as shown here plumbum remote machines

1
2from plumbum import SshMachine
rem = SshMachine("hostname", user = "john", keyfile = "/path/to/idrsa")

Creating the actual droplet is simple since digitalocean has python binding to their api digitalocean-api.But unfortunately when I attempted to ssh into one of my freshly created droplets it would not connect. The problem is I needed to add the knownhosts file. While I was researching this I found this webpage Managing Multiple SSH Keys. So I decided to make a python script which would generate the required files as described on the webpage. While I was using the script in ipython notebook, kept on getting confused which VPS I was logging into, what images where on the server etc. I happened to have some code kicking around which will dump a python dictionary into an HTML table and displays it in the ipython notebook. So I decided to use that to dump which droplets are active, what images are available etc. To convert the python object into a dictionary I just fed my function the dict property.

Basically I have 3 files:

• ipython_helpers.py - contains functions to display python dictionaries as tables. I just happened to have this kicking around. The file is on my gist here. ipython_helpers.py
• secret.py - a file which needs to contain three variables.digitalOceanToken,IdRsaPub and idRsa. Set these as appropriate for your digitalocean account.
• common.py - contains common functions, I need to refactor more. If I have to work on other IAAS frameworks besides digitalocean, I will refactor more. common.py.
• digitalocean_config.py - the file that I does everything I talked about. digitalocean_config.py
• and ofcourse last but not least an ipython notebook which gives an example of it in operation.Ipython demo notebook

Summary

This is just a short quick post on how to make tools which save you time. Before running the code, make sure you backup your .ssh directory. Also it is good to keep old code lying around, the ipython_helpers.py was from another project.

Digitalocean also is great for testing new ideas. digitalocean. So I wanted to start using mongodb, since it seemed to be similar to zodb, but I didn’t want the hassle of installing it on my system. So I just created a mongodb on digitalocean and within minutes I was able to test and try out mongodb.

These are the possible pitfalls when using Selenium Requests

• When selenium requests spawns the extra selenium windows to get the cookies this takes time.
• When doing repeated XMLHttpRequest calls, it does not seem to update the cookies, in certain cases it does not work.
• On rare occasions it can crash the headless browser.

Selenium requests is a great addon to to python selenium, and I use it frequently. It is in the cases when Selenium Requests does not work that I am talking about.

Recently a client asked me to scrape results from kickstarter. In this case he wanted all kickstarter projects which are card games in the USA. I was able to locate the XMLHttpRequest, but is sets an anti-xss token on one of the headers. I located where the token is in the webpage and extracted it, but when I set the headers, the request would fail. But when I called the request from within firebug, it worked fine. Inclued in this post is a ipython notebook which explains the technique I used to solve the problem. IPython notebook demonstrating the technique

The important function in example is in SearchPage.doajaxresultsrequest with this piece of javascript code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18var url = arguments[0];
window._jsonResult = null;
var token = document.querySelector('meta[name = "csrf-token"]').getAttribute("content");

var xmlhttp = new XMLHttpRequest();

xmlhttp.onreadystatechange = function() {
if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
    window._jsonResult = jsonResult = JSON.parse(xmlhttp.responseText);
    }}

xmlhttp.open("GET", url, true);
xmlhttp.setRequestHeader("X-CSRF-Token",token);
xmlhttp.setRequestHeader("X-Requested-With","XMLHttpRequest")
xmlhttp.setRequestHeader("Accept","application/json, text/javascript, */*; q=0.01")
xmlhttp.send();

return true;;

The url is pulled off the arguments array, the xss token is extracted from the document,and then the XMLHTTPRequest call is setup and called. I decided not to wait for the XMLHTTPRequest call to complete, but instead store the result in window._jsonResult.

A second call is then made with getajaxresult to return the result. In this case the XMLHttpRequest is parsed into json before it is stored, that means when the result is fetched by python there is no need to parse it with JSON.

The first example uses the browser class of my docker_rpyc module . The second example uses the worker class of my rpyc_docker module which means the browser is running totally headless and isolated within a docker container. Since my library uses rpyc the code is almost identical, and the same SearchPage class in the same namespace can be used. It is also possible to run multiple isolated headless browsers.

Possible improvements on the technique.

• Instead of making two calls to get the result, wait till the XMLHttpRequest finishes before returning, I am not really sure if this is an improvement.
• Have the XMLHttpRequest call save the status of the request instead of just saving the result.
• Have an array which stores XMLHttpRequests, and then repeatedly call the request function. Since javascript is able to do asycronous requests this would probably be a lot quicker for multiple XMLHttpRequests.
• A similar technique can be used to hook HMLHttpRequests, and then intercepting the results.

Why this matters ?

In the past scraping web pages could be done with just curl and raw http requests. But websites have transformed from a series of static pages, into applications running inside a web browser. As a result new web scraping techniques need to be developed. From my experience the hybrid method of web scraping websites is highly effective. So a combination of raw http calls, running headless browsers and injecting javacripts.

What is docker ?

Docker is built on top of linux namespaces, which are similar to bsd jails. Linux namespaces allow processes to be isolated in the linux kernel, each process with its own environment. Initially I attempted to use linux namespaces directly to isolate my processes, this is extremely complicated since each namespace requires its own file system and network to be configured.

Fortunately I discovered docker so I did not need to reinvent the wheel. Docker allows you to create extremely light wieght processes, each one running in its own customized linux environment. So your server could be running linux 14.04 but inside your docker container you could be running the latest version of redhat. It is best to check out the docker website to see what all of the fuss is. With Docker it is extremely simple to customize new virtual machines and the processes inside them run with no extra overhead.

What is rpyc python ?

Rpyc solves the problem on how to access python objects transparently inside a running docker container. The docker container is a running virtual machine, rpyc acts as a bridge to get around the isolation problem.

Rpyc is a remote procedure call library similar to pyro or twisted spread. All three libraries allow you to perform rpc calls on python objects, and all three libraries are great libraries. Rpyc is designed for controlling machines which you own, and it is symetric. Which twisted spread and pyro are more designed for a client to server type model. In these two libraries it is assumed that you do not control the client, and possibly the client might be hostile.

Rpyc when running in classic mode makes the remote object appear as if it is locally on your machine. If you are unfamiliar with rpyc and classic mode read Introduction to Classic RPyC. With the remote object you can do things like monkey patching, if the object crashes you can get a remote traceback or even enter into a remote debugging session. This makes development a lot easier.

Rpyc also comes with other utilities like the ability to upload or download files to your remote machine.

Introducting Rpyc_docker

Rpyc_docker is a python library which runs an rpyc server inside a docker container. There are instructions on how to setup and use rpyc_docker and rudimentary api documentation here which I am still working on Rpyc_docker and there is a github repository.

Rpyc_docker works well with the page object design pattern of testing and craping websites. For development purposes it is best to design with the browser visible and not running in a container. Later you can run multiple headless browsers each in its own docker container isolated and seperated.

The solution is to parse the data that is needed inside the browser using javascript, and then to pass the results back to python selenium using the selenium function execute_script

The function execute_script if passed arguments will attempt to convert them from python to javascript, when returning the arguments the reverse will happen.

Here is an example where two arguments are passed to the function from python.

1
2
3
4
5
6
7
8#driver is a webdriver instance

js_function = """
return (function(a,b) {
return a + b;
})(arguments[0],arguments[1]);
"""
driver.execute_script(js_function,1,2)

The example I have used is scrapping the results from duckduckgo. I created a simple example here in an ipython notebook here Inject javascript into python selenium.

The class DuckDuckGoResults takes a driver as an argument. Here is a brief description of the member functions.

• search(self,searchTerm) - takes a search term and inputs it into duckduckgo, should return when the search page results are there.
• scroll_botton(self) - utility function to scroll the page to the bottom. Used to load all the results from duckduckgo.
• load_all_results(self) - keeps on scrolling the page down until all of the results have been loaded.
• parse_resultElm(self,resultElm) - utility function for parsing a result div. Extracts the title,href and description from the div and returns it as dict.
• def get_results_python(self) - returns all of the results of the page as an array of dicts. This function is written using python. This function is extremely slow.
• def get_results_javascript(self): same as get_results_python except it uses javascript this function returns the results almost instantly.

The get_results_javascript function is an example of extracting results using javascript inside the browser and then passing them back to python selenium. Here is the javascript part of the function.

1
2
3
4
5
6
7
8
9var resultElms = Array.prototype.slice.call(document.querySelectorAll("#links>div.results_links_deep"));
return resultElms.map(function(resultElm) {
    var result = [];
    var resultAElm = resultElm.querySelector("a.result__a");
    result.push(["title",resultAElm.textContent]);
    result.push(["href",resultAElm.getAttribute("href")]);
    result.push(["snippet",resultElm.querySelector("div.result__snippet").textContent]);
    return result;
    });

There is one problem when returning dicts from javascript. Javascript dicts and objects are the same data type. An object returning from javascript into python will not convert properly. Otherwise there is no problem returning all basing datatypes including arrays. The solution is to return the dictionary structure as an array of arrays like I have done. So each item in the array will be [[“title”,”XXXXX”],[“href”,”XXXXX”],[snippet,”XXXX]]. Once the structure has been returned to python convert it into a dict like this.

1
2results = self.driver.execute_script(jsFunction)
return map(dict,results)

The map function applies dict to each member to convert it into an array of dicts so the output is the same as the python function.

For most of my web scrapping I use the firebug extension to assist me. Within firebug run the javascript code, and you can test it inside firebug.

This is an introduction to running javascript code from selenium. The big advantage is speed, and that javascript was designed to manipulate DOM structures.