May 2022 Status Update

Posted by Russell Keith-Magee on 1 June 2022

It's been another busy month at the hive! The progress we made in May isn't as obvious as last month - but the changes that have been made are an essential part of ensuring we have a stable foundation for future work on the BeeWare ecosystem.

What we've done

During May:

  • We updated the Apple support packages. These updates aren't just a version bump - they also added support that enables the iOS emulator to run natively on M1 devices. This required a major restructure of the way the support packages are bundled, using Apple's new XCframework library packaging format.
  • We addressed an issue with the use of ctypes on physical iOS devices. ctypes is a core part of the infrastructure we use to integrate with the iOS system libraries, so resolving this bug was critical for distributing apps. The fix was deep in the core of libFFI, and has been submitted for integration into the upstream project.
  • We released an Android support package for Python 3.10, as well as updated support packages for Python 3.7, 3.8, and 3.9. These support packages dramatically are now dramatically smaller, and result in smaller end-user applications, thanks to some compiler flag optimisations suggested by the community.
  • We began the process of adding support for Python 3.11 to the Apple and Android support packages. This is the first time BeeWare has been able to contribute prerelease testing to the CPython project. In the process, we discovered a small backwards incompatibility. Baring major incidents, we should be in a position to release BeeWare's support for Python 3.11 on the same day as the official 3.11.0 is finalised.
  • We developed an acceptance test suite for the Apple and Android support packages. This test suite tests for features in the Python core that have a history of being broken (such as modules in the standard library that have binary components, but have been incorrectly compiled or inadvertently excluded from the build). This test suite isn't yet formally integrated into the build process, but the hope is that it will be in the near future.
  • We modified Briefcase so that it is able to run the Android emulator on M1 hardware. This required a major rebuild of the way the Android SDK tooling is managed.
  • We made further improvements to the log capture tools on macOS. This ensures that the first few seconds of application logs are reliably captured, and the log stream is terminated when the app exits.
  • We started the introduction of Black as a part of BeeWare's development tooling. BeeWare's repositories have always enforced code style guidelines, and those code styles are very closely aligned with Black's choices. With Black formally moving out of Beta status at the start of this year, the time has come to start adopting Black as a formal project requirement. For now, Briefcase is the only project that has formally adopted Black, but you can expect this to continue to roll out over the other projects in BeeWare over the coming months.
  • We resolved some minor bugs that were present in the 0.3.0dev33 release of Toga. Most important of these was a bug that prevented the BeeWare tutorial running on iOS devices.

Many of these changes either began as, or are entirely the work of community members. There are also plenty of smaller PRs, bug fixes, and bug reports that are just as important, but are too numerous to list. A huge thanks to everyone who has contributed to BeeWare during May.

What's next?

In June, we'll be focussing on:

  • Publishing our roadmap for Q3 and beyond! Now that we have reliable development resources, we're in a position to make public commitments on the future direction of the project.
  • Completing the hiring process for another engineer to work on BeeWare full time. This hiring process is underway, but it's not too late to apply if you're interested. Full details of the position can be found on Greenhouse.
  • App Notarization on macOS. This is becoming an increasingly required part of macOS app publishing, as Apple cranks up their security requirements.
  • Improving Windows packaging. This includes improving the way the running app presents to the Task Manager, improving log capture, and app signing.
  • Improving Linux packaging. This includes addressing some issues with packaging binary dependencies.

Now Hiring!

Posted by Russell Keith-Magee on 9 May 2022

BeeWare is now hiring!

The position is a full time, Mid- to Senior position. You will be working full time in the Open Source group at Anaconda, on the BeeWare suite of tools. Full details of the position can be found on Greenhouse.

The position calls for an unusual combination of skills. The ideal candidate would have experience building GUI applications (especially mobile) and Python skills. However, because of the existing state of the Python ecosystem, most Python developers don't have GUI development experience, and most GUI developers don't have extensive Python experience. For that reason, if the position is interesting to you, but you don't have all the "must have" attributes - I would encourage you to apply anyway. A candidate with no GUI development experience will still considered, as long as they've got a demonstrated history of doing weird and wonderful things with Python. Similarly, a developer with deep GUI experience, but no Python experience, will also be considered.

The job location requirements are also unusual. The position is remote; the position requires that your working hours need to be compatible with UTC+8. This means candidates from Australia, South East and South Asia will be a natural fit. European candidates will need to be prepared for early morning starts. US/Canadian candidates will need to be prepared for evening work (very late evenings if you're in CST or EST timezones). Anaconda has the capacity to hire in the UK, Germany, India, Australia, US, and Canada. If you're not a resident of one of those countries, it may be possible to hire you, but it will likely require you to operate as a private contractor rather than a salaried employee.

I'm incredibly excited for what the future holds for BeeWare - if you'd like to come on this journey with me, please apply (and tell them Russell sent you)!

April 2022 Status Update

Posted by Russell Keith-Magee on 2 May 2022

One of the benefits of having the support of an organization like Anaconda is that we now have the time to perform little housekeeping tasks like publishing community updates. As a result, this is the first (of hopefully many) monthly updates letting you know what progress has been made in the BeeWare suite.

What we've done

During April:

Many of these PRs either began as, or are entirely the work of community members. There are also dozens of smaller PRs, bug fixes, and bug reports that are just as important, but are too numerous to list in a post like this. A huge thanks to everyone who has contributed to BeeWare during April.

What's next?

In May, we'll be focussing on:

  • Updating the support packages for macOS and iOS apps. Apple's transition to M1 hardware means there are new simulator architectures that require support; this, in turn, requires that we adopt Xcode's new XCFramework for the packaging libraries, which requires some fairly major changes to the way the support packages are built.
  • Updating the support packages on Android to support Python 3.10.
  • Notarization on M1. This is becoming an increasingly required part of macOS app publishing, as Apple cranks up the security requirements.

We're also expecting to start the hiring process in the coming month. We'll provide more details when we can point a full job description. If you think you'd like to make BeeWare your day job (or you know someone who might), now would be a good time to start polishing your resume.

Exciting news for the future of BeeWare

Posted by Russell Keith-Magee on 6 April 2022

The biggest challenge for any Open Source project is finding the resources needed to support development - and the BeeWare project is no exception.

To date, BeeWare has been primarily driven by the efforts of volunteers. We've been able to achieve amazing things with those volunteer efforts, demonstrating that Python is a viable language for GUI development on macOS, Windows, Linux, iOS and Android, as well as providing a proof-of-concept demonstration on Web. However, there are still lots of features that we'd like to add, and lots of ongoing maintenance that needs to be done as operating systems and hardware platforms evolve.

Over the years, we've been incredibly fortunate to have the financial support of our members. While I'm incredibly grateful for that support, it falls a long way short of even one full-time salary. We've also received grants (such as the PSF grant), and short-term contracts. These grants and contracts have enabled us to add specific (and much needed) features to BeeWare - but they don't provide resources for ongoing development and maintenance.

However, the brutal truth is that our rate of progress is severely limited by the volunteered resources at our disposal. As a result, I've spent a lot of time over the last few years thinking about how we can fund the work that needs to be done to make BeeWare the go-to solution for GUI development on desktop, mobile, and more - and do so without compromising the Open Source roots of the project.

I'm incredibly excited to announce that I've found a solution.

On March 21, I joined Anaconda as a Principal Engineer and Team Lead - and the team I will be leading will be focused 100% on BeeWare.

Anaconda is no stranger to Open Source. In addition to indirectly funding a number of community efforts through their Anaconda Dividend program, they directly employ people to work on Numba, Dask, Pyston and more. BeeWare will be joining this stable of projects.

Anaconda is supporting BeeWare in this way because they see the potential for Python as a language for GUI development - especially on mobile platforms. Python has established itself as an incredibly popular language in education, science, business and more - and in all those markets, there is potential to develop apps. By adding mobile platform support and app development to the list of tasks to which Python is already incredibly capable, Anaconda hopes to ensure the long term prospects of Python as a language, as well as ensuring a healthy market for the products they sell commercially.

BeeWare will continue to be a 100% Open Source project, with all contributions welcome from the community. The only difference is that Anaconda will be directly employing some of the contributors to the project. The conversations I've had with Anaconda's engineering leadership indicates they are acutely aware of the tension that exists between corporate funding and the technical direction of an Open Source project. They've indicated that they want to ensure that BeeWare's technical direction remains independent in both appearance and substance. This includes financial independence - the financial support provided by our members will remain vitally important, as it provides project-level funding for resources such as hosting and promotional materials.

The related good news is that part of my job title is "Team Lead" - so in the very near future, we'll be looking for people to join me on my team. I'm already working out the details of the team that I'd like to have, so watch this space for hiring announcements.

Anaconda has a long history of understanding and contributing to both the Python ecosystem and the Open Source community. Adding BeeWare to their stable of supported projects has enormous potential for BeeWare, and the Python ecosystem as a whole. I'm incredibly excited for what the future holds.

Some new ways to discuss BeeWare

Posted by Russell Keith-Magee on 1 May 2021

Today, the BeeWare project is making 2 changes to the way we organize our community.

Firstly, we're going to move our real-time chat community from Gitter to Discord.

When we originally set up Gitter, it was a strong option for a chat community - especially one that was focussed on a software project, due to its tight integration with Github. However, over the years, Gitter has undergone multiple changes of ownership, but has remained essentially unchanged - in some cases, it's gotten appreciably worse (their native iOS and Android clients are effectively deprecated). In the meantime, Discord has emerged as a dominant player for community chat. It has a robust mobile client, and really good tools for community moderation, as well as the option of hosting video and voice chats - something we hope to be able to use for coordinating virtual sprints.

To join the new BeeWare Discord server, visit this link.

Secondly, we're going to start using Github Discussions.

For many years, we've tried to use Github Issues as a pseudo-forum. There's a lot of similarity between how a forum works and how Github displays and manages issues. Each "issue" is effectively a forum thread, and each comment is a message in that thread. However, there are also some important differences. Issues get closed when they're resolved; forum threads, however, don't necessarily have a resolution. Most importantly, "closing" an issue can look like someone is trying to kill discussion - when all they're really trying to do is keep the list of issues manageable.

The good news is that Github has noticed the similarity too, and has launched Github Discussions. Discussions offers a familiar "forum" interface, but with the familiar Github message interface, and tight integration with Github Issues and pull requests.

We've enabled Discussions on all the key BeeWare projects - Toga, Briefcase, Rubicon ObjC, Rubicon Java, Colosseum, Travertino and Podium.

Naturally, all of these new community spaces - along with all existing BeeWare spaces - are subject to the BeeWare Community Code of Conduct.

We hope these two changes will make it easier to participate in the BeeWare community. See you there!

Release the Android Bees!

Posted by Russell Keith-Magee on 6 July 2020

9 months ago, we announced that the BeeWare Project was the recipient of a PSF Education Grant. In late 2019, we choose Asheesh Laroia to do the work. Today, we're proud to announce that BeeWare's Android support has reached a point where it is ready for mass consumption.

The BeeWare tutorial has been updated to include an Android track. The tutorial walks you through the process of writing your first app, and deploying it - first as a desktop app, and then on your phone. All you need is a Python install; everything else you need is downloaded automatically by the BeeWare tools as part of the development process.

And, to prove that it's real - we have uploaded Travel Tips to the Google Play Store. This is a Python app, in the Google Play Store, deployed from the same source code as the version in the iOS App Store.

Huge thanks go to Asheesh for all the work he's put into this project. Delivering Android support in BeeWare has required considerable technical skill, attention to detail, and creative hacks; and Asheesh has consistently delivered. Huge thanks also go to the Education Grants Committee of the Python Software Foundation for the faith they placed in BeeWare when they funded this project. In 6 months (while simultaneously negotiating a world pandemic), we've gone from a nothing to a full Android implementation. Without their financial support, this project would still be a dream.

There's still plenty of work to be done, though. There are still a lot of widgets that need Android implementations, and new widgets that we want to add. We'd like to add support for device features like cameras, GPS and accelerometers. We'd like to make it easier to use binary Python packages like NumPy, Keras, and more. And we'd like to merge the work that we've done into CPython itself.

All of those are significant projects in themselves, and will require effort comparable to that required to add Android as a supported platform. And so, we're looking for the financial support to make that happen. We're applying for new grants as opportunities arise, but the most helpful source of funds are the ongoing funds that come from memberships. Ongoing funding means we can focus on improving BeeWare, rather than chasing grants. It means being able to hire permanent staff, rather than offering short term contracts. And it means being able to make long term plans and promises to the community.

If this project proves anything, it's that money makes things happen. If you're excited by the prospects of Python on mobile platforms, please consider joining the BeeWare Project as a financial member. Or, if you've got experience with commercialisation of Open Source projects, or any other ideas for how we could please get in touch.

Python on Android: Its alive!

Posted by Russell Keith-Magee on 29 February 2020

For the last couple of months, we've had a contractor (Asheesh Laroia) working on fixing Android support in the BeeWare suite of tools.

I've incredibly happy to announce that we've just hit our first major milestone: a working pure-Python application, running on an Android device!

This isn't the end of the project - it's just the beginning. There's still lots of fine tuning to be done (especially on the size of the support libraries), and we need to integrate this support into Briefcase and Toga.

However, in the meantime, if you're adventurous, you can take Asheesh's work-in-progress for a spin. His Python Android Support repository contains the current state of the work, and includes fairly comprehensive instructions for getting started. You'll need to know at least a little bit about native Android programming to make full use of this repository at the moment; but if you want to replicate the results from the video, that repository (and the other repositories it links to) should have everything you need.

Huge thanks go once again to the Python Software Foundation. Without their financial support, this work would still be on the drawing board. This project is just one of many ways that the PSF uses donations to improve the Python community and ecosystem. If your company uses Python in any capacity, I strongly encourage you to contribute financially to the PSF so that they can continue to fund worthwhile projects like this.

Huge thanks also go to Asheesh. Without his remarkable talents, attention to detail, and delight for obscure compiler errors, we wouldn't have made the incredibly fast progress that we've seen.

Stay tuned for more announcements soon!

We have a contractor for our Android contract!

Posted by Russell Keith-Magee on 26 November 2019

A couple of months ago, we announced that the BeeWare project had received a grant from the PSF to improve our support for Android. At that time, we issued a call for contractors to help us complete this work.

We're very happy to announce that we've now selected a contractor: Asheesh Laroia.

Asheesh is a regular speaker at Python events, where he has delved into a range of detailed and complex topics. He also impressed us with the list of unconventional engineering integration projects he's been involved with in a professional and casual capacity.

When asked why he applied to work with BeeWare on this contract, Asheesh said: "I use an Android phone every day, and I'm honored to be able to help implement the BeeWare vision of using Python to build first-class, native applications."

Asheesh will be starting work in mid December, and if all goes well, we should start seeing significant results by mid to late February. If you'd like to keep track of progress, you can follow BeeWare on Twitter; we'll also post larger updates on this blog.

BeeWare Project Awarded a PSF Education Grant

Posted by Russell Keith-Magee on 25 September 2019

The BeeWare Project wants to make it possible for all Python developers to write native apps for desktop and mobile platforms. We have solid support for most desktop operating systems and iOS, but we know our Android support is lacking. The BeeWare core team knows what needs to be done to address the problem - what we’ve been missing are time and resources.

Thanks to the PSF Education Grants group, that’s no longer an issue. We’ve been awarded a US$50,000 grant to bring BeeWare’s Android support to a level comparable with our iOS support. We currently don’t have the time to do the work ourselves, so we’re calling for contractors to help us deliver this support.

This is a paid contract, which we anticipate lasting 3-6 months (depending on the experience of the winning contractor). You don’t have to be based in the US or Europe, either; the opportunity is open to anyone who can meet the requirements of the contract.

Unfortunately, the task calls for some sophisticated skills, and we’re not in a position to provide extensive mentoring. A successful bid is likely to require some prior experience, and a history with the technologies involved.

A full role description and scope of work for the contract is available. To register your interest, please forward your resume and cover email to contracts@beeware.org.

We look forward to being able to announce full Android support in the near future!

2018 Google Summer of Code - VOC Optimization

Posted by Patience Shyu on 14 August 2018

Google Summer of Code is coming to an end. I've spent the summer working on optimizing the VOC compiler, and I’m super excited to share the results.

Results

There are a couple of ways to evaluate the performance improvement from my project.

Microbenchmarks

Firstly, we introduced a microbenchmarking suite. Each microbenchmark is a small piece of Python code that tests a single and specific Python construct, or datatype, or control flow. The benchmarking infrastructure itself is crude (essentially it just tells you the total amount of processor time it took to run, with no fancy statistics) but it has been extremely useful to me while working on performance features to verify performance gain.

The idea is that the benchmarking suite is not to be run as part of the full test suite, but rather as needed and manually whenever an optimization is implemented. It also provides a way to check and prevent performance regression, especially on the "optimized" parts of VOC. While it doesn't really make sense to record specific numbers, as they will always vary from machine to machine, it should be reasonably easy to compare two versions of VOC. Benchmark numbers are included on each optimization-related PR I've worked on this summer (see PR log below), and I hope that more benchmarks will be added as more performance efforts are carried out in the future.

Pystone

Pystone is a Python Dhrystone, a standard benchmark for testing the performance of Python on a machine. Here are the before and after results on my machine:

May 10th, 2018:

$ python setup.py test -s tests.test_pystone test_pystone (tests.test_pystone.PystoneTest) ... Pystone(1.2) time for 50000 passes = 101.833 This machine benchmarks at 490.998 pystones/second

$ python setup.py test -s tests.test_pystone test_pystone (tests.test_pystone.PystoneTest) ... Pystone(1.2) time for 50000 passes = 101.298 This machine benchmarks at 493.595 pystones/second

$ python setup.py test -s tests.test_pystone test_pystone (tests.test_pystone.PystoneTest) ... Pystone(1.2) time for 50000 passes = 102.247 This machine benchmarks at 489.014 pystones/second

On current master (Aug 14th, 2018):

$ python setup.py test -s tests.test_pystone test_pystone (tests.test_pystone.PystoneTest) ... Pystone(1.2) time for 50000 passes = 11.2300 This machine benchmarks at 4452.37 pystones/second

$ python setup.py test -s tests.test_pystone test_pystone (tests.test_pystone.PystoneTest) ... Pystone(1.2) time for 50000 passes = 10.9833 This machine benchmarks at 4552.36 pystones/second

$ python setup.py test -s tests.test_pystone pystone (tests.test_pystone.PystoneTest) ... Pystone(1.2) time for 50000 passes = 10.9498 This machine benchmarks at 4566.29 pystones/second

Conclusions

Some things that I learned about VOC while working on this project:

1. Object creation in the JVM is expensive. This definitely does not mean that the VOC user writing Python should think about minimizing the number of objects that she creates, but rather that any time we can non-trivially reduce the number of objects created during bytecode transpilation or in VOC-defined function calls, we can expect to see a huge performance boost. Integer and boolean preallocation, which is about reusing objects that have already been created, was one of the most significant improvements we made this summer.

2. Method calls in VOC are expensive. This is essentially due to the process of invoking a callable: you have to check that the method is defined on the object, then construct it (read: object creation!), and check the arguments, before it can actually be called. (This is done using reflection, which is super interesting and confusing in itself.) And this is the reason why refactoring the Python comparison functions made such a big performance impact, because we were able to circumvent this process.

3. Exception-heavy code is expensive. Again, this is not to say that the programmer is on the hook for being frugal when throwing exceptions, but that VOC benefits greatly by avoiding the use of exceptions internally except when strictly necessary. For instance, Python uses StopIteration exceptions to signal the end of a for loop, and they quickly rack up when you have nested loops (everything is ultimately related to object creation!). That was the motivation for the nested loops optimization.

If I may be a bit more reflective here, one of the a-ha! moments I had this summer was realizing that to really optimize something, you have to understand where its biggest problems are first. I remember pitching to Russ at the start of the summer things like loop unrolling, constant folding, even converting to SSA-form (you know, stuff I heard about optimzation in my compilers class) and he was saying to me, think simpler. While working on my project, I used a profiler to understand exactly which parts of VOC were slow, and that information drove the changes we implemented. I think it worked out pretty well!

Future Work

  • Minimize boxing of primitive types like String and Int. As VOC is written half in Python, half in Java, a single integer can be found in various representations on its way through the compiler -- as a Python object, unboxed to a primitive Java int, then packaged back up to a Python object. This problem was (somewhat incoherently) addressed in my proposal, but ultimately we couldn't come up with a good abstraction to support it.
  • Build a peephole optimizer. CPython's peephole optimizer scans generated bytecode to identify sequences of bytecode that can be optimized, VOC could benefit from this too.
  • Hook up more benchmarks, which serve as both proof of the kinds of programs VOC can currently compile and areas ripe for performance improvement.

Thank you

I will wrap this up by giving big thanks to Russ, my mentor. The time you spent helping me form my ideas, patiently answering my questions and reviewing my work was invaluable to me. It couldn't have been easy keeping up with what I was doing especially since I started improvising halfway through the summer. I am so grateful for your help, thank you.

RSS Feed

Archive