Mega cloud storage client, megacmd

I have been recently working on a command line client for Mega cloud storage service. If you are not familiar with mega, let me give you a brief intro. Mega is a storage service founded by Kim Dotcom exactly one year after MegaUpload was raided by US government and seized his entire property. The case against him was that he hosted an upload service which was being misused by people and become the source of piracy. He recently came up with a nice storage solution out of New Zealand which encrypts data at client side and transfers to the server. At mega server side, nobody will be able to decrypt your data since it is encrypted at client side. Your password is the encryption key used for the data. The downside is that, once you forget the password you will not be able to reset your password or recover you data back.

I started using Mega from day it was launched and was very much impressed by its service. The major factor that impressed me was speed and free storage space. When all the major cloud storage service providers provide 5Gigs of max free quota. Mega provides 50Gigs of quota, which is awesome. The another part of awesomeness is the speed. I compared download and upload speed by downloading same files in Google drive, dropbox, etc. Mega is around 8X faster than all of them. Mega supports multiple parallel connections for download and upload, thereby helps to increase speed further. Currently, Mega.co.nz is a client web application written in Javascript and runs completely out of browser. Chrome and Firefox works perfect for mega. Just signup using just email and password and get started in 2 mins.

Mega provides an open API service for clients to manage mega account. The current Javascript implementation makes use of APIs to perform all actions on mega account within the browser. They have a limited bare minimum developer documentation available at developers page. I loved the security aspect, storage quota and speed perspective of Mega. But I wanted a command-line client to access Mega, which I found missing. I am used to s3cmd utility, which is a command-line utility to perform file transfers to Amazon S3. I started writing a utility called megacmd in Golang. Go is a statically typed, at the same time highly flexible interesting language that I have been hacking around for sometime. Go is C like language that provides concurrency, synchronization by communication and garbage collection as part of the language. Having looked at AWS S3 protocol, Mega is more complicated to implement than S3 client. S3 provides a simple object store APi to provide object path and value to be stored. But, mega does more sophisticated things like keeping the filesystem tree as metadata.

I will run through a very quick overview of how a client interacts with Mega service and its architecture.

Mega uses json a the unit of API request and response. It uses the following request and response formats:

Request: { a : command type, [argument : value]* }.
Response: [res1,res2,...] or error code -errcode or [-errcode]

A user has to initialize a user session by login with username and password. A login request consists of username and a custom hash(password). It responds with sessionId, RSA private key and user master key. All of them encrypted. User master key is a symmetric encryption key for decrypting keys specific to all the files stored by a user account. In order to decrypt the sessionId, first compute the password hash, and decrypt the master key using the password hash. Once master key is decrypted, use that to decrypt the RSA composite key. Composite RSA key contains p,q and d components (Read RSA Algorithm Operation), all of them encoded in multi-precision integer format. Once you decode these components, we can decrypt the sessionId. SessionId is also represented as multi precision int (m). Hence, sessionId = (m ^ d) % n where n = p x q. Since mega uses json for request and response, it should be ascii encoded. Hence, all binary data is represented as a specialized base64url encoding with padding characters (=) removed. Hence sessionId is converted into base64url encoding. Once we decode sessionId, it used used for all the mega file operations. For identifying the requests, mega requires a id to be passed along with requests which needs to be incremented everytime. Client requests are post to the url https://eu.api.mega.co.nz/cs?id=seqid&sid=sessId. Mega uses 128 bit block AES encryption all over the place.

Mega stores files in node hierarchical parent-child model like standard filesystems. Mega has three root nodes, either Mega Cloud, Trash or Inbox. Each node can be either a folder type, file type or root. Mega stores each node as a block of bytes with a size, timestamp, key, attribute, parent, node type identified by a hash. Attribute contains encrypted filename by using encrypted file key. User master key is used to decrypt the file key. Using the file key, file content and attribute can be decrypted. The file key is a composite key which contains additional initialization vector and meta-CBC-MAC (Message Authentication Code) data in case of a file node. The actual file content is encrypted in AES counter mode using this init vector. CBC-MAC meta data is used for integrity verification of the file.

Once you get a session ID, we can use file list APi to download meta data about all the nodes in the filesystem tree. The response contain, one metadata entry per node.

To download a file, we should send a request with node Hash. The response will contain a url at which we can download the file. The file should be downloaded in chunks of size, 128K / 384K / 768K / 1280K / 1920K / 2688K / 3584K / 4608K / … (every 1024 KB). This is to allow integrity checks per chunk. While downloading each chunk, it should be decrypted using AES-Counter mode using the file key.

To upload a file, we should generate a random key to be used as file key and initialization vector. An upload request with file size is issued and it returns a response with a url at which we can do uploads. Uploads are done in chunks as download by posting encrypted data at http://uploadurl/chunkstart-chunkend. Once all the uploads are complete, we receive a completion handle. We should post a file upload completion request with completion handle, meta-MAC, attributes, parent hash, etc to finalize the upload.

There are other operations like delete, move, update attribute, etc available.

Since we keep a local copy of entire filesystem, we need a way to update our local cache of filesystem when some other client using the same account updates the filesystem by adding, removing or updating files or folders. For eg, if we login to mega.co.nz account in two browsers with same credential, we can see that if we make any change on one browser that gets reflected in other as well. In order to provide filesystem change notification, mega provides a mechanism for server to talk to the client.

The client should hit an API url, https://eu.api.mega.co.nz/sc?sn=Sn. Sn is received from this API call or from list filesystem API call. This API will respond with a wait url or a list of file node operations. If it provides a wait url. We should perform a get request to that wait url. This get request will block until a filesystem event occurs. Once it returns, it should query the API to get the list of events and perform those events in the local filesystem copy to synchronize with the filesystem at the server side. Since documentation is minimum, I had to download the javascript implementation and had to make out the working myself reading code.

About megacmd utility

Mega service provides a mechanism to access the Mega FS tree in a model with hash identifiers. But they are not readable and we need a path name representation using resource URIs. So I initially wrote a library for go called go-mega which provides Mega API abstraction in terms of hash and filesystem tree. Then, I wrote another go package called megaclient which abstracts in terms of file path and it provides similar operations as s3cmd. Code for both can be found in my github.

Repositories:

go mega megacmd

megacmd is a command-line utility written on top of megaclient package. The usage for megacmd is as follows:

Usage ./megacmd:
megacmd [OPTIONS] list mega:/foo/bar/
megacmd [OPTIONS] get mega:/foo/file.txt /tmp/
megacmd [OPTIONS] put /tmp/hello.txt mega:/bar/
megacmd [OPTIONS] delete mega:/foo/bar
megacmd [OPTIONS] mkdir mega:/foo/bar
megacmd [OPTIONS] move mega:/foo/file.txt mega:/bar/foo.txt
megacmd [OPTIONS] sync mega:/foo/ /tmp/foo/
megacmd [OPTIONS] sync /tmp/foo mega:/foo

-conf="/Users/slakshman/.megacmd.json": Config file path
-force=false: Force hard delete or overwrite
-help=false: Help
-ignore-same-size=false: Consider files with same size and path suffix as same
-recursive=false: Recursive listing
-verbose=1: Verbose
-version=false: Version

Downloadable binaries for GNU/Linux and Mac OSX are available.

Please read more and find documentation at https://github.com/t3rm1n4l/megacmd.

If you find any bugs or would like to contribute to the project, please feel free to file an issue or send a pull request at github.