August 2022 Status Update

Postato da Russell Keith-Magee il 1 September 2022

The main focus of the BeeWare team during August was binary module support on mobile platforms. Although we haven't got the final results of this work yet, we have significant progress to report for both iOS and Android. We've also had a number of other improvements land over the course of the month.

What we've done

During August:

What's next?

In September, we hope to wrap up the work on binary dependencies. On Android, this is mostly a matter of completing the work to support multiple Python versions. On iOS, we need to develop the tools to manage the compilation of third-party binary libraries. We have an initial proof of concept of this; we need to finish these tools, and develop the infrastructure to distribute. We're also hoping to start conversations with the Anaconda team about Conda integration with Briefcase, and start improving GUI testing in Toga.

July 2022 Status Update

Postato da Russell Keith-Magee il 1 August 2022

Another month of important updates to the BeeWare project! This month, our focus was on packaging improvements on Linux and Windows.

In addition the technical progress, we welcomed Malcolm Smith (@mhsmith on Github) to the Anaconda BeeWare team! Malcolm brings his considerable experience developing and maintaining Chaquopy, a set of tools and libraries for building Android applications that use Python. Chaquopy tackles the "Python on Android" problem from the perspective of adding Python to an existing Android Studio project rather than writing apps entirely with Python; as a result, it features much tighter integration with native Android Studio tooling. One particularly interesting feature of Chaquopy is that it supports binary dependencies on Android - a key feature that BeeWare's Android tooling currently lacks. Historically, Chaquopy was a closed source tool with licenses available for open source projects; however, as a result of joining Anaconda, Malcolm has released Chaquopy as an Open Source project!

What we've done

During July:

  • We added support for Linuxdeploy plugins for Linux AppImages. These plugins are an important part of making AppImages self-contained, as many libraries need to provide the Linuxdeploy tooling hints to help find all the resources needed at runtime. Adding support for these plugins should address most of the issues we've been seeing with moving AppImages between Linux versions.
  • We added a Flatpak backend for Linux packaging. Flatpak is an alternative packaging format to AppImage which has the support of a number of large Linux distributions. For now, AppImage continues to be the default Linux packaging format, but we may change this default in the future.
  • We improved the packaging of Windows apps by adding a stub binary. We're still hunting some bugs in this stub binary, but when those are resolved, the Windows apps generated by Briefcase will present to the operating system with consistent app naming and icons. The stub binary also enables us to catch runtime problems with the app in a way that makes diagnosing app problems much easier.
  • We presented a webinar about native application development. If you weren't able to attend the webinar live, a recording is available; register here to watch.
  • We got an initial proof of concept of a Toga app running on Android using Chaquopy as a base, accessing a binary library (matplotlib), deployed with Briefcase. Over the coming weeks, we're hoping this proof of concept will evolve into major improvement to BeeWare's Android tooling.
  • We modified Briefcase to use OS-appropriate directories for caching. While the ~/.briefcase location has served us well, it isn't a location that adheres to platform-native conventions for storing app resources, so we've moved Briefcase's cache of downloads, tools and templates to a platform-appropriate location.
  • We addressed an issue with the current working directory leaking into the runtime path. This was especially common with Linux apps, but it could potentially be an issue with other platforms as well.
  • We've started the process of making Toga widget APIs internally consistent. While we've aimed to keep naming of widgets' attributes and handlers consistent, there are a couple of annoying inconsistencies. We've started the process of cleaning up these inconsistencies.
  • We've started looking into binary dependencies on iOS apps.

What's next?

In August, our major focus will be binary dependencies on mobile. This is one of the biggest missing pieces of the BeeWare story at present. Being able to utilize Chaquopy as a base gives us a huge head start on Android, so it seems likely we'll have a working solution for Android in the near future. However, on iOS, we're still at the "we don't know what we don't know" stage. It's unclear if we'll have a fully working solution by the end of the month, but we should have at least some progress to report.

June 2022 Status Update

Postato da Russell Keith-Magee il 4 July 2022

It's been another busy month at the hive! This month, we've had a couple of high profile improvements, and some slow progress towards longer term goals. These improvements have been primarily in Briefcase and app packaging, but there have been some minor Toga improvements as well.

What we've done

During June:

What's next?

In July, we'll be focussing on:

  • Completing the work on Linux and Windows application packaging.
  • Presenting a webinar about native application development. If you'd like to attend, registration for this webinar is open
  • Starting investigations into binary module support on mobile platforms.

We're also delighted to announce that we've successfully completed our hiring process; so, in 2 weeks, there will be another Bee in the hive! I'm very excited about the possibilities that this additional team member will afford. Look out for a welcome announcement in the coming weeks.

2022Q3 Roadmap

Postato da Russell Keith-Magee il 24 June 2022

One of the benefits of having the support of an organisation like Anaconda is that we are now in a position to make more concrete commitments on project goals, and the likely timeframe when key features will be delivered. To that end, in addition to monthly status updates, we're going to start publishing regular updates on our project roadmap so the community has a better idea what to expect over the coming months.

Project estimation is difficult at the best of times; and project estimation on a FLOSS project is especially complex. In addition to any work that we are planning internally as a team, we need to manage contributions from the community. If a community member contributes a piece of work that isn't on our roadmap, we need to review that work. As a result, our ability to deliver on our own roadmap is dependent on how much is contributed by the community - something that is very hard to predict ahead of time.

In addition, several of the topics listed on this roadmap are complex, foundational tasks, and potentially contain unknown complications. As with any complex work, it can be difficult to predict ahead of time how long that work will take.

As a result, this roadmap should be read as a guide to what we aim to focus on over the coming quarter, rather than a hard commitment of features that will be made available on a specific deadline.

Q3 priorities

During Q3, we're planning to work on:

Linux packaging

Breifcase's AppImage packaging backend has had a cluster of issues reported related to inclusion of third party libraries, and the post-processing that needs to be performed on an AppImage. We've already started work on this during Q2; early in Q3, you should expect to see this work wrapped up. This will include:

  • Adding support for linuxdeploy plugins for our existing AppImage backend; and
  • Adding a Flatpak backend.

Windows packaging

Briefcase has an MSI packaging backend for Windows; and while this backend works, it doesn't deliver an end-user experience that is as polished as we'd like. We should be starting initial work on Windows packaging in the coming week, with the complete solution completed in early Q3. This will include:

  • Adding a stub binary for Windows apps, so that Briefcase apps present to the task manager with the apps name and icon, rather than "pythonw";
  • Improving log capture on Windows, so that it is easier to diagnose when problems occur; and
  • Fixing some issues with the way the site is defined and used. This may also address some issues we've seen with binary modules in some Windows apps.

Binary modules

The biggest caveat on BeeWare's mobile app story has always been that we don't provide support for binary packages on mobile platforms. In Q3, we're going to start addressing this limitation.

There are already tools like crossenv that can simplify this process, and we've got a PR in our review queue that provides a proof-of-concept implementation of Android binary packaging. We need to take these tools, plus other tools in the Python ecosystem, and extend Briefcase so that we can present a clear development story for mobile developers that want to use modules with binary components.

On iOS, there is an additional complication - we need to resolve how to manage dynamic loading of modules. BeeWare's iOS support currently relies on static compilation of all binary modules. This was originally done because prior to iOS8, dynamic linking was prohibited on iOS. The days of iOS8 are now long behind us; but the Apple support package has retained this old approach, mostly because it wasn't clear what approach Apple would allow through the App Store review process. We will be making changes to Briefcase's iOS support to accomodate dynamic module loading, and then extending that support to include user-provided binary modules.

Testing (especially GUI testing)

Without testing, it's very difficult to ensure the stability and reliability of code in the long term. Unfortunately, many of the areas where BeeWare operates are inherently difficult to test. Visual GUI tests are notoriously difficult to perform; packaging tests require the ability to actually install and run finished applications; testing mobile applications requires access to mobile device hardware in a CI setup. As a project, we've avoided writing tests for many of our core features due to these complexities - and we've paid the price for this in regressions. Now that we have more time, we can focus on setting up the frameworks to let us perform this testing.

This includes adding:

  • Support package tests, ensuring that the packages include valid libraries;
  • App template tests, ensuring that Briefcase is able to generate valid and installable applications; and
  • GUI tests, ensuring that Toga generates apps that look and behave the way they should.

Testing is an ongoing process, so it's highly unlikely we'll have "100% test coverage" by the end of the quarter. The goal will be to add the frameworks that allow us to add tests at all, so that we can retrofit tests onto existing code, and add tests on all new code going forward. Ideally, these tests will be fully automated tests running in CI. However, we may accept semi-automated or manual tests as a short term of interim measure, especially if those tests lay the groundwork for fully automated tests in the future.

Conda integration

Briefcase is currently uses PyPI and pip as source of Python packages. However, PyPI and pip isn't the only packaging solution in the Python ecosystem - another commonly used packaging source is Conda.

At this point, I must recognise the elephant in the room - Anaconda, the company that produced and maintains Conda, is my employer. However, I must stress that the fact this is on our roadmap is not as a result of a directive from my employer. The original proposal for this idea predates even my earliest hiring discussions with Anaconda. It is being pursued now for three reasons:

  1. The key value of Conda is that it simplifies the process of delivering complex binary packages, and ensuring that the binary dependencies for those packages are isolated from any packages that might be present on the user's computer. This core problem statement shares a lot of similarities with the core problem associated with application packaging. Conda also has much more nuanced representation of hardware platforms, alternatives for binary artefacts, and options for custom repositories. There is a possibility that using Conda as a packaging base may simplify some of the problems we've seen with distributing binary dependencies.
  2. There is a significant portion of the Python community that have commercial needs that requires the use of the managed repositories provided by Anaconda, or are simply more familiar with Conda tooling than PyPI/pip.
  3. As a result of having 1 (soon to be 2) full time engineers, we now have the resources to explore complex changes that might not have otherwise been explored. We are exploring this idea because we now have the resources to do so; we were not provided those resources on the condition that we explore this idea.

We don't anticipate dropping support for pip, or for PyPI as a package source. We're looking at extending Briefcase to provide another commonly used and requested option for packaging, rather than replacing PyPI/pip.

Longer term goals

Looking beyond Q3, the big items on the horizon are:

Upstreaming

The Apple and Android support packages maintained by BeeWare include a number of patches against the core CPython. We want to begin the process of getting these modifications into the CPython codebase.

There have been two major impediments to this in the past:

  1. Development resources When we were a volunteer driven project, we didn't have the resources to engage meaningfully with the CPython core team to shepherd our patches into the CPython codebase.
  2. Testing Supported platforms in the CPython codebase need to have CI solutions.

We now have resources, so we can dedicate the time needed to upstream patches; and once we've completed our testing work in Q3, we'll be in a position to recommend CI configurations to the CPython core team. That means the work of upstreaming patches and making iOS and Android fully supported platforms can begin.

Toga 1.0

The broad goal for "Toga 1.0" could be considered "Tkinter, but with some additional bells to show off". We want to deliver a core widget set of the "essentials", plus some nice features to show off, like WebViews and camera or GPS integration. The exact set of features for Toga 1.0 is still a subject for discussion; I'd expect to see a more concrete feature list and plan emerge in Q4.

Summary

As with most of the work that has been done in Q2, the foundational work we're planning for Q3 is not as glamorous as adding high profile GUI widgets, but it's all necessary work if we're going to provide a stable and reliable development story. Once this foundation is established, we'll be in a much better position to rapidly develop new features that are visible - and to ensure that those features are stable and reliable as BeeWare develops.

May 2022 Status Update

Postato da Russell Keith-Magee il 1 June 2022

It's been another busy month at the hive! The progress we made in May isn't as obvious as last month - but the changes that have been made are an essential part of ensuring we have a stable foundation for future work on the BeeWare ecosystem.

What we've done

During May:

  • We updated the Apple support packages. These updates aren't just a version bump - they also added support that enables the iOS emulator to run natively on M1 devices. This required a major restructure of the way the support packages are bundled, using Apple's new XCframework library packaging format.
  • We addressed an issue with the use of ctypes on physical iOS devices. ctypes is a core part of the infrastructure we use to integrate with the iOS system libraries, so resolving this bug was critical for distributing apps. The fix was deep in the core of libFFI, and has been submitted for integration into the upstream project.
  • We released an Android support package for Python 3.10, as well as updated support packages for Python 3.7, 3.8, and 3.9. These support packages dramatically are now dramatically smaller, and result in smaller end-user applications, thanks to some compiler flag optimisations suggested by the community.
  • We began the process of adding support for Python 3.11 to the Apple and Android support packages. This is the first time BeeWare has been able to contribute prerelease testing to the CPython project. In the process, we discovered a small backwards incompatibility. Baring major incidents, we should be in a position to release BeeWare's support for Python 3.11 on the same day as the official 3.11.0 is finalised.
  • We developed an acceptance test suite for the Apple and Android support packages. This test suite tests for features in the Python core that have a history of being broken (such as modules in the standard library that have binary components, but have been incorrectly compiled or inadvertently excluded from the build). This test suite isn't yet formally integrated into the build process, but the hope is that it will be in the near future.
  • We modified Briefcase so that it is able to run the Android emulator on M1 hardware. This required a major rebuild of the way the Android SDK tooling is managed.
  • We made further improvements to the log capture tools on macOS. This ensures that the first few seconds of application logs are reliably captured, and the log stream is terminated when the app exits.
  • We started the introduction of Black as a part of BeeWare's development tooling. BeeWare's repositories have always enforced code style guidelines, and those code styles are very closely aligned with Black's choices. With Black formally moving out of Beta status at the start of this year, the time has come to start adopting Black as a formal project requirement. For now, Briefcase is the only project that has formally adopted Black, but you can expect this to continue to roll out over the other projects in BeeWare over the coming months.
  • We resolved some minor bugs that were present in the 0.3.0dev33 release of Toga. Most important of these was a bug that prevented the BeeWare tutorial running on iOS devices.

Many of these changes either began as, or are entirely the work of community members. There are also plenty of smaller PRs, bug fixes, and bug reports that are just as important, but are too numerous to list. A huge thanks to everyone who has contributed to BeeWare during May.

What's next?

In June, we'll be focussing on:

  • Publishing our roadmap for Q3 and beyond! Now that we have reliable development resources, we're in a position to make public commitments on the future direction of the project.
  • Completing the hiring process for another engineer to work on BeeWare full time. This hiring process is underway, but it's not too late to apply if you're interested. Full details of the position can be found on Greenhouse.
  • App Notarization on macOS. This is becoming an increasingly required part of macOS app publishing, as Apple cranks up their security requirements.
  • Improving Windows packaging. This includes improving the way the running app presents to the Task Manager, improving log capture, and app signing.
  • Improving Linux packaging. This includes addressing some issues with packaging binary dependencies.

Now Hiring!

Postato da Russell Keith-Magee il 9 May 2022

BeeWare is now hiring!

The position is a full time, Mid- to Senior position. You will be working full time in the Open Source group at Anaconda, on the BeeWare suite of tools. Full details of the position can be found on Greenhouse.

The position calls for an unusual combination of skills. The ideal candidate would have experience building GUI applications (especially mobile) and Python skills. However, because of the existing state of the Python ecosystem, most Python developers don't have GUI development experience, and most GUI developers don't have extensive Python experience. For that reason, if the position is interesting to you, but you don't have all the "must have" attributes - I would encourage you to apply anyway. A candidate with no GUI development experience will still considered, as long as they've got a demonstrated history of doing weird and wonderful things with Python. Similarly, a developer with deep GUI experience, but no Python experience, will also be considered.

The job location requirements are also unusual. The position is remote; the position requires that your working hours need to be compatible with UTC+8. This means candidates from Australia, South East and South Asia will be a natural fit. European candidates will need to be prepared for early morning starts. US/Canadian candidates will need to be prepared for evening work (very late evenings if you're in CST or EST timezones). Anaconda has the capacity to hire in the UK, Germany, India, Australia, US, and Canada. If you're not a resident of one of those countries, it may be possible to hire you, but it will likely require you to operate as a private contractor rather than a salaried employee.

I'm incredibly excited for what the future holds for BeeWare - if you'd like to come on this journey with me, please apply (and tell them Russell sent you)!

April 2022 Status Update

Postato da Russell Keith-Magee il 2 May 2022

One of the benefits of having the support of an organization like Anaconda is that we now have the time to perform little housekeeping tasks like publishing community updates. As a result, this is the first (of hopefully many) monthly updates letting you know what progress has been made in the BeeWare suite.

What we've done

During April:

Many of these PRs either began as, or are entirely the work of community members. There are also dozens of smaller PRs, bug fixes, and bug reports that are just as important, but are too numerous to list in a post like this. A huge thanks to everyone who has contributed to BeeWare during April.

What's next?

In May, we'll be focussing on:

  • Updating the support packages for macOS and iOS apps. Apple's transition to M1 hardware means there are new simulator architectures that require support; this, in turn, requires that we adopt Xcode's new XCFramework for the packaging libraries, which requires some fairly major changes to the way the support packages are built.
  • Updating the support packages on Android to support Python 3.10.
  • Notarization on M1. This is becoming an increasingly required part of macOS app publishing, as Apple cranks up the security requirements.

We're also expecting to start the hiring process in the coming month. We'll provide more details when we can point a full job description. If you think you'd like to make BeeWare your day job (or you know someone who might), now would be a good time to start polishing your resume.

Exciting news for the future of BeeWare

Postato da Russell Keith-Magee il 6 April 2022

The biggest challenge for any Open Source project is finding the resources needed to support development - and the BeeWare project is no exception.

To date, BeeWare has been primarily driven by the efforts of volunteers. We've been able to achieve amazing things with those volunteer efforts, demonstrating that Python is a viable language for GUI development on macOS, Windows, Linux, iOS and Android, as well as providing a proof-of-concept demonstration on Web. However, there are still lots of features that we'd like to add, and lots of ongoing maintenance that needs to be done as operating systems and hardware platforms evolve.

Over the years, we've been incredibly fortunate to have the financial support of our members. While I'm incredibly grateful for that support, it falls a long way short of even one full-time salary. We've also received grants (such as the PSF grant), and short-term contracts. These grants and contracts have enabled us to add specific (and much needed) features to BeeWare - but they don't provide resources for ongoing development and maintenance.

However, the brutal truth is that our rate of progress is severely limited by the volunteered resources at our disposal. As a result, I've spent a lot of time over the last few years thinking about how we can fund the work that needs to be done to make BeeWare the go-to solution for GUI development on desktop, mobile, and more - and do so without compromising the Open Source roots of the project.

I'm incredibly excited to announce that I've found a solution.

On March 21, I joined Anaconda as a Principal Engineer and Team Lead - and the team I will be leading will be focused 100% on BeeWare.

Anaconda is no stranger to Open Source. In addition to indirectly funding a number of community efforts through their Anaconda Dividend program, they directly employ people to work on Numba, Dask, Pyston and more. BeeWare will be joining this stable of projects.

Anaconda is supporting BeeWare in this way because they see the potential for Python as a language for GUI development - especially on mobile platforms. Python has established itself as an incredibly popular language in education, science, business and more - and in all those markets, there is potential to develop apps. By adding mobile platform support and app development to the list of tasks to which Python is already incredibly capable, Anaconda hopes to ensure the long term prospects of Python as a language, as well as ensuring a healthy market for the products they sell commercially.

BeeWare will continue to be a 100% Open Source project, with all contributions welcome from the community. The only difference is that Anaconda will be directly employing some of the contributors to the project. The conversations I've had with Anaconda's engineering leadership indicates they are acutely aware of the tension that exists between corporate funding and the technical direction of an Open Source project. They've indicated that they want to ensure that BeeWare's technical direction remains independent in both appearance and substance. This includes financial independence - the financial support provided by our members will remain vitally important, as it provides project-level funding for resources such as hosting and promotional materials.

The related good news is that part of my job title is "Team Lead" - so in the very near future, we'll be looking for people to join me on my team. I'm already working out the details of the team that I'd like to have, so watch this space for hiring announcements.

Anaconda has a long history of understanding and contributing to both the Python ecosystem and the Open Source community. Adding BeeWare to their stable of supported projects has enormous potential for BeeWare, and the Python ecosystem as a whole. I'm incredibly excited for what the future holds.

Some new ways to discuss BeeWare

Postato da Russell Keith-Magee il 1 May 2021

Today, the BeeWare project is making 2 changes to the way we organize our community.

Firstly, we're going to move our real-time chat community from Gitter to Discord.

When we originally set up Gitter, it was a strong option for a chat community - especially one that was focussed on a software project, due to its tight integration with Github. However, over the years, Gitter has undergone multiple changes of ownership, but has remained essentially unchanged - in some cases, it's gotten appreciably worse (their native iOS and Android clients are effectively deprecated). In the meantime, Discord has emerged as a dominant player for community chat. It has a robust mobile client, and really good tools for community moderation, as well as the option of hosting video and voice chats - something we hope to be able to use for coordinating virtual sprints.

To join the new BeeWare Discord server, visit this link.

Secondly, we're going to start using Github Discussions.

For many years, we've tried to use Github Issues as a pseudo-forum. There's a lot of similarity between how a forum works and how Github displays and manages issues. Each "issue" is effectively a forum thread, and each comment is a message in that thread. However, there are also some important differences. Issues get closed when they're resolved; forum threads, however, don't necessarily have a resolution. Most importantly, "closing" an issue can look like someone is trying to kill discussion - when all they're really trying to do is keep the list of issues manageable.

The good news is that Github has noticed the similarity too, and has launched Github Discussions. Discussions offers a familiar "forum" interface, but with the familiar Github message interface, and tight integration with Github Issues and pull requests.

We've enabled Discussions on all the key BeeWare projects - Toga, Briefcase, Rubicon ObjC, Rubicon Java, Colosseum, Travertino and Podium.

Naturally, all of these new community spaces - along with all existing BeeWare spaces - are subject to the BeeWare Community Code of Conduct.

We hope these two changes will make it easier to participate in the BeeWare community. See you there!

Release the Android Bees!

Postato da Russell Keith-Magee il 6 July 2020

9 months ago, we announced that the BeeWare Project was the recipient of a PSF Education Grant. In late 2019, we choose Asheesh Laroia to do the work. Today, we're proud to announce that BeeWare's Android support has reached a point where it is ready for mass consumption.

The BeeWare tutorial has been updated to include an Android track. The tutorial walks you through the process of writing your first app, and deploying it - first as a desktop app, and then on your phone. All you need is a Python install; everything else you need is downloaded automatically by the BeeWare tools as part of the development process.

And, to prove that it's real - we have uploaded Travel Tips to the Google Play Store. This is a Python app, in the Google Play Store, deployed from the same source code as the version in the iOS App Store.

Huge thanks go to Asheesh for all the work he's put into this project. Delivering Android support in BeeWare has required considerable technical skill, attention to detail, and creative hacks; and Asheesh has consistently delivered. Huge thanks also go to the Education Grants Committee of the Python Software Foundation for the faith they placed in BeeWare when they funded this project. In 6 months (while simultaneously negotiating a world pandemic), we've gone from a nothing to a full Android implementation. Without their financial support, this project would still be a dream.

There's still plenty of work to be done, though. There are still a lot of widgets that need Android implementations, and new widgets that we want to add. We'd like to add support for device features like cameras, GPS and accelerometers. We'd like to make it easier to use binary Python packages like NumPy, Keras, and more. And we'd like to merge the work that we've done into CPython itself.

All of those are significant projects in themselves, and will require effort comparable to that required to add Android as a supported platform. And so, we're looking for the financial support to make that happen. We're applying for new grants as opportunities arise, but the most helpful source of funds are the ongoing funds that come from memberships. Ongoing funding means we can focus on improving BeeWare, rather than chasing grants. It means being able to hire permanent staff, rather than offering short term contracts. And it means being able to make long term plans and promises to the community.

If this project proves anything, it's that money makes things happen. If you're excited by the prospects of Python on mobile platforms, please consider joining the BeeWare Project as a financial member. Or, if you've got experience with commercialisation of Open Source projects, or any other ideas for how we could please get in touch.

RSS Feed

Archive