The Buzz

2018 Google Summer of Code - Implement asyncio support in VOC

Posted by Yap Boon Peng on 11 August 2018

In the blink of an eye, Google Summer of Code (GSoC) 2018 has come to an end. During the three months long coding period, I have contributed several patches in VOC repository of BeeWare, all working towards the ultimate end goal of running asyncio module in VOC. In this blog post (which is my first actual blog post by the way 😄), I will document what I have done so far, why I couldn't make it to the end goal (yea, unfortunately I couldn't get asyncio to work at the end of GSoC 2018), and what's left that needs to be done in order to achieve the end goal (or at least make part of asyncio work).

Project Spotlight: Colosseum

Posted by Russell Keith-Magee on 6 October 2017

This article was originally published on the BeeWare Enthusiasts mailing list. If you'd like to receive regular updates about the BeeWare project, why not subscribe?

When you're designing a GUI app - be it for desktop, mobile, or browser - one of the most fundamental tasks is describing how to lay widgets out the screen. Most widget toolkits will use a grid or box packing model of some kind to solve this problem. These models tend to be relatively easy to get started, but rapidly fall apart when you have complex layout needs - or when you have layouts that need to adapt to different screen sizes.

Instead of inventing a new grid or box model, the Toga widget toolkit takes a different approach, using a well known scheme for laying out content: Cascading Style Sheets, or CSS. Although CSS is best known for specifying layout in web pages, there's nothing inherently web specific about it. At the end of the day, it's a system for describing the layout of a hierarchical collection of content nodes. However, to date, every implementation of CSS is bound to a browser, so the perception is that CSS is a browser-specific standard.

That's where Colosseum comes in. Colosseum is a browser independent implementation of a CSS rendering engine. It takes a tree of content "nodes" - such as a DOM from a HTML document - an applies CSS styling instructions to layout those nodes as boxes on the screen. In the case of Toga, instead of laying out <div> and <span> elements, you lay out Box and Button objects. This allows you to specify incredibly complex, adaptive layouts for Toga applications.

But Colosseum as a project has many other possible uses. It could be used anywhere that there is a need for describing layout outside a browser context. For example, Colosseum could be the cornerstone of a HTML to PDF renderer that doesn't require the involvement of a browser. It could also be used as a test harness and reference implementation for the CSS specification itself, providing a lightweight way to encode and test proposed changes to the specification.

The current implementation is based on Facebook's yoga project - it was originally a line-for-line port of yoga's javascript codebase into Python. However, yoga only implements the Flexbox portion of the CSS3 specification.

This week, we started a big project: rewriting Colosseum to be a fully standard-compliant CSS engine. The work so far can be found in the globe branch of the colosseum repository on Github. The first goal is CSS2.1 compliance, with an implementation of the traditional CSS box model and flow layout. Once we've got a reasonable implementation of that, we'll look to adding Grid and FlexBox layouts from the CSS3 specification set.

This is obviously a big job. CSS is a big specification, so there's a lot of work to be done - but that also means there's lots of places to contribute! Pick a paragraph of the CSS specification, build some test cases that demonstrate the cases described in that paragraph, and submit a patch implementing that behaviour!

It also highlights why your financial support is so important. While we could do this entirely with volunteered effort, we're going to make much faster progress if a small group of people could focus on this project full time. Financial support would allow up to significantly ramp up the development speed of Colosseum, and the rest of the BeeWare suite.

If you would like to see Colosseum and the rest of BeeWare develop to the point where it can be used for commercial applications, please consider supporting BeeWare financially. And if you have any leads for larger potential sources of funding, please get in touch.

2017 Google Summer of Code - Port Cricket to use Toga, instead of Tkinter

Posted by Dayanne Fernandes on 25 August 2017

After almost 4 months of work on Google Summer of Code 2017, finally I'm completing my proposal. Every widget migration and every commit/PR/issue/discussion with my mentors about Cricket , Toga and rubicon-objc were detailed on the Issue 58.

2017 Google Summer of Code - Batavia improvements

Posted by Adam Boniecki on 23 August 2017

With Google Summer of Code 2017 program nearing its end, it is time to summarize what I got done during the summer working on Batavia.

Batavia is a part of BeeWare's collection of projects. As it is still in its early stage of development, for my part I offered to implement a number of features missing from Batavia, ranging from elemental data types, through JSON manipulation and language constructs such as generators. I posted my proposal in this GitHub thread and kept it updated with my progress on a weekly basis.

Note that by the end of GSoC, we have decided to diverge from the proposal and forgo implementation of contextlib in favor of support for Python 3.6 2-byte wide opcodes.

Overall it was great learning experience and fun. Big thanks to my mentors Russell Keith-Magee and Katie McLaughlin, and the whole BeeWare community.

2017 Google Summer of Code - Testing Toga/Settings API

Posted by Jonas Schell on 22 August 2017

Google Summer of Code 2017 is coming to an end. After three month of working on the BeeWare project, I would like to summarise my work and share my experiences.

A new yak for the herd: BeeKeeper

Posted by Russell Keith-Magee on 31 July 2017

Testing is a skill that is a vital part of every programmers training. Learning how to write good tests helps you write more robust code, and ensures that when you've written code that works, it keeps working long into the future. It can also help you write better code in the first place. It turns out that well architected code, with high cohesion and low coupling, is also easy to test - so writing code that is easy to test will almost always result in better overall code quality.

An important step in "levelling up" your testing experience is to start using a Continuous Integration, or CI service. A CI service is a tool that automatically runs your test suite every time someone makes a change - or every time someone proposes a change in the form of a pull request. Using a CI service makes sure that your code always passes your test suite - you can't accidentally slip in a change that breaks a test, because you'll get a big red warning notification.

CI is such an important service that many companies exist solely to provide CI-as-a-service. The BeeWare project has, for various projects, used TravisCI and CircleCI. Both these tools provide free tiers for open source projects, and have generously sponsored BeeWare with capacity upgrades at various times.

However, the BeeWare has had an interesting relationship with commercial CI services. This is for two reasons.

Firstly, our test suites - especially for VOC and Batavia - take a long time to run. These two projects require tests that repeatedly start and shut down virtual machines (for Java and JavaScript, respectively), and no matter how much you optimize the code being tested, the startup and shutdown time of a virtual machine eventually adds up. We also need to run our test suites on multiple versions of Python - at present, we support Python 3.4, 3.5 and 3.6, with 3.7 on the horizon. There are also subtle changes in micro versions that may require testing.

We've been able to speed up the duration of a test run by splitting up the test suite and running parts of the suite in parallel, but that forces us up against the second problem. Commercial CI services usually operate on a subscription model; higher subscriptions providing more simultaneous resources. However, our usage pattern is highly unusual. Most of the year, we get a slow trickle of pull requests that require testing. However, a couple of times a year, we have a large sprint, and we have a flood of contributions over a short period of time. At PyCon US, we have had groups of 40 people submitting patches - and they all need their submissions tested by CI. And time is a factor - the sprints only last a couple of days, so a fast turnaround on testing is essential.

If we were to subscribe to the top tier subscription levels of CircleCi and TravisCI, we still wouldn't have enough resources to support a sprint - but we'd be massively overresourced for the rest of the year. We'd also have to pay $750 or more a month for this service, which is a budget we can't afford.

So - we had a problem. To run our test suite effectively, we needed massively parallelized resources to run a test suite quickly; and at certain times of the year, we need extremely large numbers of these resources.

We also had other automated tasks that we wanted to perform. We wanted to do code linting (automated checking of code style) before a PR was tested. We wanted to check spelling of documentation. And we wanted these tasks to feed back into GitHub as automated comments and specific code review status markers, informing contributors not just that a problem occurred - but what problem occurred, and where in their code.

We also wanted to manage pipeline builds - there's no point in doing a full test of multiple versions of Python once you've established the tests are failing on one version. And there's no point testing at all if there are code style problems.

We also wanted to do things that weren't just testing. We wanted to check that contributor agreements have been signed. We wanted to automate deployment of websites and documentation.

What we had wasn't just a CI problem. It was a problem where we wanted to automatically run arbitrary code, in a safe way, in response to a GitHub event.

I've been trying to find a CI service that can meet our needs for over a year. But over the last year, a few thoughts started to congeal in my head.

• Amazon provides a API (EC2) that allows you to spool up machines of varying complexity (up to 64 CPUs, with almost 500GB of RAM), and pay by the minute for that usage.
• Docker provides the tools for configuring, launching, and running code in an isolated fashion
• Amazon also provides an API (ECS) to control the execution of Docker containers.

There's nothing specific about AWS EC2 or ECS either - you could just as easily use Linode and Kubernetes, or Docker Swarm, or Microsoft Azure... you just need to have the ability to easily spool up machines and run a Docker container. After all: a test suite is just a Docker container that runs a script that starts your test suite. A linting check is a Docker container that runs a script that lints your code. A contributor agreement check is a Docker container that checks the metadata associated with a pull request.

All you need then is a website that can receive GitHub event notifications, and start Docker containers in response.

At the start of July, I found myself between jobs, and uttered the fateful question "How hard could it be?" And so, today, I'm announcing BeeKeeper - BeeWare's own CI service.

BeeKeeper deploys as a Heroku website, written using Django. After configuring it with Github and AWS credentials, it listens for Github webhooks. When a Pull Request or Push is detected, BeeKeeper creates a build request; that build request inspects the code in the repository looking for a beekeeper.yml configuration file. That configuration file describes the pipeline of tasks that will be performed, and for each task, the type of machine that should be used, any environment variables that are required, and the Docker image that will be used.

BeeKeeper also allows the site admin to describe what resources will be used to satisfy builds. A task can say it needs a "High CPU" instance; but the BeeKeeper instance can determine what "High CPU" means - is it 4 CPUs or 32? When those machines are spooled up, how long will they be allowed to sit idle before being shut down again? How many machines should be sitting in the pool permanently? And what is the upper limit on machines that will be started?

A companion tool to BeeKeeper is Waggle. Waggle is a tool that prepares a local definition of a task so it can be used by BeeKeeper - it compiles the Docker image, and uploads it into ECS so that it can be referenced by tasks. (It's called "Waggle" because when worker bees discover a good source of nectar, they return to the hive and do a waggle dance that tells other bees how to find that source).

We've also provided a repository called Comb (named after honey comb, the place bees store all the nectar they find) that defines the task configurations that a BeeKeeper instance can use. We've provided some simple definitions as part of the base Comb repository; each BeeKeeper deployment should have one of these repositories of it's own.

There's still a lot of work to do, but we're already using BeeKeeper to Batavia and VOC, and the upcoming PyCon AU sprints will be our first outing under high-load conditions. Some back-of-envelope calculations predict that for around $50, we'll be able to provide enough CPU resources for each test run to complete running in 10 minutes or less, supporting a sprint of dozens of people.

Although BeeKeeper was written to meet the needs of the BeeWare project, it's an open source tool available for anyone to use. If you'd like to take BeeKeeper for a spin, come along to the sprints, or check out the code on GitHub.

BeeKeeper is also an example of the sort of product you'd see more of if BeeWare development was funded full time. I was able to build BeeKeeper because I had a spare couple of weeks between jobs. There is no end to the tools and libraries like BeeKeeper and Waggle that could be built to support the software development process - all that is missing is the resources needed to develop those tools. If you'd like to see more tools like BeeKeeper in the world, please consider joining the BeeWare project as a financial member. Every little bit helps, and if we can reach a critical mass of supporters, I'll be able to start working on BeeWare full time.

A request for your help

Posted by Russell Keith-Magee on 5 April 2017

Around 4 years ago, I made the first commit to Cricket - the first tool that would eventually become part of the BeeWare suite. Since then, the BeeWare project has grown to encompass support for mobile platforms, two alternate Python implementations, and a cross platform widget set - as well as the developer tools that started the project originally.

For most of this time, BeeWare has been a volunteer effort. Initially, it was a solo project; however, over the last year in particular, the number of people who have contributed to BeeWare has grown rapidly. Over 300 people have now made contributions to the various BeeWare tools and libraries - due, at least in part, to the Challenge Coins we've been offering at sprints.

Of particular note is the team of 7 people who have joined the project as apiarists, helping share the load of project maintainence. I can't thank these people enough - without their assistance, I wouldn't have made anywhere near as much progress over the last year.

You may have noticed that over the last few months, progress has been especially rapid. This is because, for the last six months, BeeWare development has been partially funded by very accomodating employers at Jambon Software. My contract with Jambon allowed me to spend significant periods of time being paid to work on BeeWare - and, not surprisingly, it was possible to make enormous progress as a result. The last 6 months has seen:

• Extensive improvements to Batavia and VOC;
• An Android backend for Toga;
• A Django backend for Toga, enabling Toga apps to be deployed as web apps;
• A Winforms backend for Toga, enabling Toga apps to run on Windows with a modern appearance;

Unfortunately, my contract with Jambon is coming to a close - which means my contributions to BeeWare will go back to being what my spare time allows.

This also means the rate of progress will also slow. There's still plenty to do: there's plenty of the Python standard library to port to Batavia and VOC; Toga needs much wider widget support; Colosseum needs to be extended so it supports the full CSS box model, not just CSS3 Flexbox; and the tools that started it all - Cricket, Bugjar, Duvet, and others - all need to be ported to Toga.

I would like to be able to work on BeeWare full time. However, the simple reality is that unless I can find a way to pay for this work, it will only be able happen in my spare time.

So - this is an appeal to you - the Python community. If you are excited by the prospect of having access to Python on mobile platforms, or you would like to write applications in Python that have completely native user interfaces - I need your help.

For just US$10 a month - you can join the BeeWare project as a member, and help make this dream become a reality. If I can find 1000 people in the Python community who want these these tools and are willing to support their development financially, I can start working on BeeWare full time. Of course, this target is even easier if companies get involved and sponsor at higher tiers.

If I can find more than 1000 people, then even more becomes possible. The obvious option would be to hire other experienced developers to assist with the work. The idea of having others to bounce ideas off during the development process is very appealing. However, we can also use this as an opportunity to do some social good.

For some time, I've had an open offer to mentor anyone who wants to get involved with Open Source contribution using the BeeWare project. However, not many people have been able to seriously take up this offer, because the time required to seriously take up the offer is prohibitive. I'd like to be able to extend my offer to more than just a casual mentoring relationship. I'd like to be able to hire - and pay - one or more junior developers for the BeeWare project, and focus on giving that opportunity to people from underrepresented demographics.

It's still early days for BeeWare. Financial support means faster progress. More widgets. Better documentation. More of everything you’ve seen so far from BeeWare. If I can find full time funding for myself - or better still, for myself and a small team - then I have no doubt that the BeeWare suite will become a viable alternative for commercial projects in very short order. Best of all, we will be able to do this without having to give up on the ideals of the open source movement.

I'm excited for what the future holds for BeeWare. I hope you'll join me on this journey.

(And if you’re contemplating signing up, and you’re coming to PyCon US in Portland this May, let me drop a gentle hint… sign up now. It will be worth it #cryptic)

Come sprint with us at PyCon US 2017

Posted by Katie McLaughlin on 1 February 2017

The tickets for PyCon US 2017 have been given away. We look forward to seeing everyone who can make it to the conference at Booth 103!

---

Want to to go @PyCon, can’t afford it? @PyBeeWare has 2 tickets to give away. Email contact@beeware.org and tell us why you want to bee there!

— PyBee (@PyBeeWare) January 30, 2017

---

PyCon US 2017 is running in Portland, Oregon from May 17 - 25, and it's bound to be another amazing conference.

For the second year in a row, the BeeWare team will be on site with a booth in the Exhibit Hall, along with other Open Source projects from the Python world.

With this booth, we get two tickets to the conference. This includes:

• access to the opening reception
• 3 days of conference talks, expo hall / job fair
• breakfasts, breaks, lunches, and
• swag bag

Thing is, both Russell and I have already registered.

So, we want to give you the ticket.

Yes, you.

If you can get yourself to Portland on the conference days, we want to give you our free ticket.

What do we want in return?

Just a little bit of your time.

The Bee Team will be helping to staff our booth, but we'd also like to see (and give!) talks, so helping us out by running the booth would be lovely. (Plus, and I'm sure Russell would agree, just helping out on the booth earns you a coin)

Plus, we'd love for you to stick around for the famous coding Sprints. These are held in the four days after the event, while people are still in town. We get coffee, lunch, and a room full of tables, chairs and copious amount of power points, and we code on projects. We'll be running a BeeWare sprint where we will be mentoring and helping first time contributors earn their shiny challenge coin

Does this sound like something you'd be interested in?

Please, email us!

Tell us about yourself! Who you are, what you do, why you want to go to PyCon and what makes you interested in Python.

We need to allocate our tickets early, so please email us by February 12, 2017

If you have any questions, just ask myself or Russell!

We'd love to see you there! ✨

[This article has been cross-posted on glasnt.com/blog]

Money, Money, Money

Posted by Russell Keith-Magee on 10 December 2016

At PyCon AU 2015, and again at DjangoCon US 2015, I gave a talk entitled "Money Money Money: Writing software, in a rich (wo)man's world". The talk was a summary of the issues around one of the biggest problems that I see facing the open source community: how to provide the resources that are needed to develop and maintain the software that we, as a community depend upon. This means providing maintenance and support for established projects, large and small; but also providing an ecosystem where new ideas can be incubated, developed and matured until they present compelling alternatives or significant benefits over closed source offerings.

It's been almost 18 months since I first presented this talk, but the issue persists. I haven't been alone in noticing and drawing attention to this issue, either. Nadia Eghbal was commissioned to write a white paper for the Ford Foundation entitled Roads and Bridges highlighting the chronic need for resources to support the basic infrastructure that underpins large parts of the modern economy. Eric Holscher (maintainer of Read the Docs) blogged about the problems he's had raising funds, despite the fact that the service he delivers is a widely used - arguably indispensible - part of the Python ecosystem.

However, despite this attention, it still doesn't get anywhere near as much attention as it should. And it's an issue that is of pressing importance to me, as the BeeWare project is looking for ways to fund the development needed to take us from "interesting technical demo" to "compelling technical solution".

A few months back, it was suggested that I should publish a blog post to accompany the video presentation. I dragged my feet on doing this, until industrious BeeWare contributor and all-around nice guy Elias Dorneles offered to take my speakers notes and convert it into a transcript.

So - here it is. Money Money Money: Writing software, in a rich (wo)man's world. If you've got any questions, disagreements, requests to present this at your conference, or just a generic offer of a bag of cash, you can reach me at russell@keith-magee.com.

Hello Website

Posted by Russell Keith-Magee on 4 July 2016

Willkommen auf der neuen Website des BeeWare-Projekts!

Die ursprüngliche BeeWare-Website wurde vor ein paar Jahren geschrieben, als BeeWare noch ein sehr experimentelles Projekt war. Die alte Website war eine einzelne Seite, wobei die einzelnen Projekte ihre eigene Web-Identität behielten. Im Laufe der Zeit ist die Zahl der Mitwirkenden gewachsen, die Zahl der Unterprojekte ist gewachsen, und die Zahl der Mitglieder des Kernteams hat sich verdreifacht.

Als das BeeWare-Projekt an Komplexität zunahm, konnte die alte Website nicht mehr mit den Anforderungen des Projekts Schritt halten, so dass das Kernteam eine Auszeit von der Arbeit an BeeWare selbst nahm und die Website neu schrieb.

Die neue Website verwendet Lektor, ein statisches Website-Generator-Werkzeug, das von dem bekannten Pythonista Armin Ronacher geschrieben wurde. Dies erlaubt uns, die Website weiterhin unter Verwendung von Github Pages zu hosten, aber gibt uns die Flexibilität, eine leichte Datenbank für die vielen Arten von Inhalten zu definieren, die wir verwalten müssen.

Obwohl wir die Website heute eröffnen, ist sie noch nicht vollständig. Es gibt Inhalte, die noch geschrieben werden müssen, und ein Erscheinungsbild, das noch festgelegt werden muss. Wir sind jedoch der Meinung, dass die neue Website besser ist als die alte, und deshalb lohnt es sich nicht, den Start noch länger hinauszuzögern.

Dies gibt auch Ihnen - der BeeWare Community - eine großartige Gelegenheit, einen Beitrag zu leisten. Wenn Sie etwas finden, das auf der Seite fehlt, oder etwas, das Ihrer Meinung nach besser ausgedrückt oder gestaltet werden könnte, ist die gesamte Seite auf GitHub verfügbar. "Forken" Sie das Repository, nehmen Sie die Änderung vor, von der Sie glauben, dass sie notwendig ist, und reichen Sie ein "Pull-Request" ein (gegen den Lektor-Zweig - nicht gegen den Master). Alle Beiträge sind willkommen, und wie bei allen Beiträgen zu BeeWare ist auch bei Beiträgen zur Website der Einsender berechtigt, eine exklusive BeeWare Challenge Coin zu erhalten.

Wir hoffen, dass Ihnen die neue Website gefällt; wenn Sie Feedback haben, lassen Sie es uns das mit einen Ticket oder über unseren Gitter-Kanal wissen.