November 2022 Status Update
November has brought some very important improvements to BeeWare. Although these changes don't have an immediate impact for users, they're going to form a vital part of our work going forward.
What we've done
During November:
- We added a test mode to Briefcase. This is a huge step forward, as it means we are now able to run automated tests for iOS and Android, as well as validation tests for packaged app behaviors. As a proof of concept, we've already begun porting our support package validation app - this is an app that we've historically needed to run manually.
- We added the ability for an AppImage to customise the Dockerfile used to build the project. This was required to support the use of Rust-based dependencies (such as cryptography) in AppImages; however, the approach we've taken should allow for any other customisation that may be required, without the need to maintain a fork of the AppImage project template.
- We've added the ability for users to add customisations to the Android Gradle file for their projects. As with the Dockerfile changes, this allows for small customisations without the need to use a completely forked template.
- We improved the resilience of Briefcase on flaky network connections. Previously, an interrupted download of a support package could leave Briefcase in a broken state; this should no longer be possible.
- We've improved type hinting in the Briefcase codebase, especially relating to the ToolCache class.
- We've improved the handling of Android apps that fail immediately on app startup. Previously, if your app crashed due to something like a Python syntax error, it was possible for Briefcase to hang in a way that made it look like the issue was due to Briefcase being unable to start the app, rather than a user code problem.
- We've corrected a problem on Windows that was preventing app output from being captured in the Briefcase log.
- We completed the process of modernising the layout and tooling of the Toga repository, including the introduction of Black, Towncrier, setuptools_scm, and a large reorganisation of the code structure
- We've begun building the first automated test of Toga's GUI behavior. This is still in the early stages, but it will form the a core part of our testing infrastructure in the near future.
- We've added preliminary support for Python 3.12 to Briefcase, Toga and Rubicon ObjC.
- We've updated the codebase of Rubicon-ObjC to use Black.
What's next?
December will be a slow month due to the Christmas and New Year break; however, until then, we'll continue to focus will continue to be on testing. We'll continue to build out the testing tools that we've worked on this month, and start building tests using those tools. We also hope to begin work on a firm 1.0 roadmap for Toga.
Want to get involved?
Want to get involved? Here's 8 open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:
- Modify the handling of app signing options
- Filter out a message generated after Xcode updates
- Add the ability to refresh the support package in a project
- Provide protection against creating projects with "semi-reserved" names
- Add the ability to configure the ABIs built by an Android project
- Add support for the ANDROID_HOME environment variable
- Add a command summary to briefcase --help
- Add an option to display the platforms supported by a command
Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.
October 2022 Status Update
How time flies! Progress in October was a little slower than in past months; preparing for and attending DjangoCon US, plus some well-earned holiday leave reduced the amount of time we had to work on new features. However, we were able to make a number of significant improvements.
What we've done
During October:
- We restored the web backend for Toga! Web deployment has always been part of the BeeWare story, but we hit some technological challenges with the approach that we were taking in the Toga 0.2 branch. However, with the addition of WASM as an officially supported platform in CPython 3.11, and the release of PyScript, we have a new way of getting Python in the browser. We have been able to use these tools to build a much improved Toga web backend.
- We added support for deploying Briefcase apps as static web sites.
- We released official support for Python 3.11 on macOS, iOS and Linux - on the day that Python 3.11 was released!. Python 3.11 support for Android and Windows is ready to go, and should be published in a day or two.
- We modified Briefcase to write its logs into subdirectory, instead of littering them in the project directory.
- We presented a talk at DjangoCon US 2022. The video for this talk should be up in around a month. A major feature of this talk was a demo of an Electron-style "web site as an app", built entirely in Python. This demo (called Positron) has been included in the Toga repository as an example.
- We attended the DjangoCon US 2022 sprints, and handed out 4 challenge coins to new contributors!
- We added the ability to retrieve widgets by ID to the Toga API.
- We made a big change to the way that Toga backends are discovered. This new approach has two major advantages - firstly, it means third-parties can register their own backends; but more importantly, it means that code no longer needs to explicitly provide a backend. This simplifies the creation of test cases - but it also means that icons, images and fonts no longer need to be "late bound". This is an internal detail, but it significantly simplifies the code using those data types.
- We added the ability to create Image objects from raw data, rather than an image file.
- We started the process of modernising the layout and tooling of the Toga repository. Some of these changes have already landed; more will come in the coming days. This ensures that we have a stable foundation for future work on Toga.
What's next?
During November, we continuing to focus on Toga, with a particular focus on testing. A particular focus will be working out how to test graphical features of Toga - an area that has historically been managed entirely with manual testing. Having strong automated testing of graphical features will be a key milestone in ensuring Toga remains stable in the long term.
Want to get involved?
Want to get involved? Here's 8 open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:
- Improve protection against corrupted or incomplete downloads
- Modify the handling of app signing options
- Filter out a message generated after Xcode updates
- Add the ability to refresh the support package in a project
- Provide protection against creating projects with "semi-reserved" names
- Add the ability to configure the ABIs built by an Android project
- Add support for the ANDROID_HOME environment variable
- Silence an warning that is displayed when an Android emulator has no skin defined
Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.
2022Q4 Roadmap
With Q3 completed, it's time to provide an update on the long term goals and priorities of the BeeWare project. As always, this roadmap should be read as a guide to what we aim to focus on over the coming quarter, rather than a hard commitment of features that will be made available on a specific deadline.
Q3 progress
We were able to hit our biggest Q3 goals:
- Linux packaging - We've investigated and addressed all the known issues with Linux packaging. There's one pending enhancement that would expand the range of third-party binary packages that can be included in Briefcase projects.
- Windows packaging - We've introduced a stub binary for Windows apps (as well as a Visual Studio target for Windows projects). This allowed us to significantly improve log capture, so when a Windows app crashes it's a lot easier to obtain diagnostic information. This change also corrected most of the issues with the way Windows apps were using the site module; however, there is still one issue related to the handling of packages that have complex post-installation processes.
- Binary modules - By far the biggest achievement of Q3. With the release of Briefcase 0.3.10, we now support binary modules on iOS and Android. We have also published 40 of the most popular binary modules, compiled for iOS and Android.
However, we didn't hit all our goals:
- Testing - Testing support has improved substantially. We have a much more thorough CI configuration on Briefcase that verifies that templates generate valid projects. We also have a testbed project that we can use to check that support packages are behaving correctly. However, we are not yet able to run this test project in CI, so we don't have automated validation of PRs against the support packages. We also made very little progress in GUI testing.
- Conda integration - We began initial discussions around Conda integration, on two fronts: firstly, to investigate the possibility of using Conda as a base for package management in apps; and secondly, about using conda-forge as a framework for managing community contributions. Both of these conversations are ongoing; it will likely take some time before these efforts bear fruit.
Q4 priorities
In Q4, we're planning to focus on:
Toga
The primary focus for Q4 will be Toga. There are a number of pieces of work involved with this:
- Housekeeping - Basic repository maintenance tasks, like enabling black, introducing pre-commit hooks, towncrier, and automating release infrastructure.
- Example consolidation - Toga's examples folder has become a sprawling collection of code that is part documentation, part testbed. We need to start consolidating these examples into 2 apps - one that is a user-facing demonstrator of Toga features, and one that can be used as a functional test of Toga features.
- Testing - Modernising and cleaning up the existing test suite, and working out how to automate GUI tests
- Documentation - Toga's documentation has lagged behind an acceptable standard for a long time,
- Planning - Developing a clear plan for what will be in Toga 1.0
- Cross-platform hardware - While it has always been technically possible to support the APIs needed to access device hardware, there hasn't been a cross-platform API for this work. In Q4, we hope to demonstrate at least 1 interesting piece of mobile device hardware.
Although we won't get to Toga v1.0 in Q4, we will land a v0.3 final, after several years of maintaining dev pre-releases as the main way to use Toga.
Web support
Toga has historically had support for the Web as a platform; however, that effort was put on hold because the approach we were taking wasn't proving viable. However, with the release of Python 3.11, WASM becomes an officially supported CPython platform. We've also seen the release of PyScript, demonstrating that client-side Python in the browser is an extremely viable proposition. The time has come to resurrect BeeWare's support for the web as a deployment platform. This will involve getting Toga's web backend into a working state, and adding a web deployment backend for Briefcase.
Longer term goals
Looking into next year, the big items on the horizon are:
Upstreaming
With binary modules now being supported on both iOS and Android, and the introduction of a Tier 3 support level in PEP11, the prospect of elevating iOS and Android to official supported platforms is seeming increasingly plausible. Our hope is to go to PyCon US next year in a position to make a concrete proposal to the CPython core team.
Toga 1.0
The broad goal for "Toga 1.0" could be considered "Tkinter, but with some additional bells to show off". We want to deliver a core widget set of the "essentials", plus some nice features to show off, like WebViews and camera or GPS integration. We hope to establish the feature list for Toga 1.0 in Q4; once that plan is in place, we'll have a better idea of when we will be able to deliver on that plan.
A Forge for mobile packages
As of today, all binary modules for mobile platforms are being maintained by the BeeWare team. This isn't a viable solution long term; we need to move to an environment where the community can manage binary packages for mobile platforms.
Summary
For the last 6 months, the focus of BeeWare has been on Briefcase and the support packages - ensuring that we have a feature complete, robust set of tools for deploying apps on any platform. In Q4, our focus is shifting to the tools we need to build apps on those platforms, and ensuring we have a stable foundation for future development. It's taken a long time for BeeWare to get this far - but we're getting very close to unlocking the remarkable potential of the Python ecosystem on mobile devices.
September 2022 Status Update
September has been a big month for the BeeWare team. There have been lots of small updates - but there's one very large, eagerly anticipated update that has taken several months to complete.
What we've done
During September:
- We added support for binary packages on mobile platforms! There's more detail related to this than can fit in a bullet point; more details below.
- We added the ability to remove files from a templated app. This can be used to trim down a project, removing parts of the Python interpreter that you don't need (such as standard library modules that you don't need)
- We've changed the way we use templates, switching to using branches based on the Briefcase release, rather than the Python release. This enables us to make development changes to templates without affecting the published stable release.
- We reworked the way Briefcase manages integrations with the tools needed to build apps. This was a mammoth project, but as a result the process of verifying tools is faster, and a number of bugs related to having stale builds have been resolved.
- We've made a number of improvements to console handling of log messages, especially on Windows
- We improved error handling in Toga when TLS1.2 or TLS1.3 isn't supported by Windows.
- We started conversations with the teams at Anaconda responsible for conda about using Conda as a base for Briefcase apps, and on using Conde-forge style infrastructure to support binary packaging. Nothing concrete has come from these conversations yet; these conversations are ongoing.
Binary package support you say?
All the new binary packaging support is contained in Briefcase 0.3.10.
On Android, binary module support was achieved by moving to Chaquopy as a base for Android projects. Along with support for binary modules, this change has the added benefit of faster startup times, and less bugs at runtime. This means you'll need to re-generate any Android project - the Python source code should work as-is, but there's no simple way to convert the Android Gradle project from the older Rubicon base to the newer Chaquopy base.
On iOS, we've reworked how the support packages are constructed, and added support for installing binary modules from a Beeware-specific package repository that includes builds of iOS binary wheels. We've also added asset signing to the iOS Xcode project - and applied the same changes to the macOS backends. Of course, this also means you'll need to re-generate any iOS project to take advantage of these changes.
While we are now able to support binary packages on mobile platforms, this doesn't mean that every binary package is automatically supported. Binary wheels need to be compiled for mobile platforms, and compiling for mobile platforms can be complicated. However, we've compiled ~40 of them most popular packages with binary components (including numpy, pandas, cryptography, pillow and matplotlib), and made them available as dependencies. In order to get working builds, some of these packages are slightly older versions (e.g., cryptography is a version from before the introduction of the rust dependency); however, we've endeavoured to provide the most recent versions possible.
A full list of supported packages can be found on the Chaquopy package repository for Android, and the BeeWare repository for iOS. If your project uses one of these packages, you only need to add them to the requires definition in your pyproject.toml, then update and re-build your project, and you're done.
If there’s a package you want that isn't on this list, you'll need to request a binary wheel. For Android, open a ticket on the Chaquopy repository. For iOS, open a ticket on this repository. If you're adventurous, you can also try building wheels yourself. The issue trackers are on a project that has as "forge-like" tool for building wheels, in the server/pypi folder; there are tools and documentation describing how to build a package. The use of these projects is a short-term measure; in the longer term, we want to move to an actual “forge”-style community-managed model for managing binary packages.
What's next?
For the last few months, the focus of the BeeWare team has been on Briefcase and the support packages. In October (and the rest of Q4), we'll be shifting our focus to Toga. The initial focus will be on improving the foundation of the project - testing and build infrastructure; but once those pieces are in place, we'll start adding new features.
We'll also be at DjangoCon US in San Diego - and there will be some web-related updates related to the presentation that will be given at that conference. We hope we'll see you there!
Want to get involved?
Want to get involved? Here's 10 open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:
- Write Briefcase logs into a subdirectory, instead of the project folder
- Improve protection against corrupted or incomplete downloads
- Modify the handling of app signing options
- Modify the Briefcase new project wizard to show which repository and branch it's using
- Filter out a message generated after Xcode updates
- Add the ability to refresh the support package in a project
- Provide protection against creating projects with "semi-reserved" names
- Add the ability to configure the ABIs built by an Android project
- Add support for the ANDROID_HOME environment variable
- Silence an warning that is displayed when an Android emulator has no skin defined
Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.
August 2022 Status Update
The main focus of the BeeWare team during August was binary module support on mobile platforms. Although we haven't got the final results of this work yet, we have significant progress to report for both iOS and Android. We've also had a number of other improvements land over the course of the month.
What we've done
During August:
- We have continued to work on the changes needed to migrate Android support to Chaquopy. This has included some major changes to the Android Gradle template, and developing a Rubicon Java compatibility layer. Eventually, Toga will migrate to use the Chaquopy APIs directly, but in the interim, the Rubicon APIs will continue to work.
- We have started updating Chaquopy to support multiple Python versions. Historically, Chaquopy has only supported a single Python3 version (3.8); for compatibility with Briefcase, we are adding support for Python 3.9 and 3.10, as well as preparing the ground for a 3.11 release when Python 3.11 is finalised.
- We have modified the Python Apple support packages to provide the binary portions of the Python standard library as dynamically loaded binary modules, and modified the iOS and macOS templates to use and sign these binary modules. These changes haven't landed yet, but they have been tested on simulators and physical devices. Along the way, we've also updated the support packages to use OpenSSL 3.
- We have modified Briefcase to make the Python support package optional, at the discretion of the project template. This was needed because Chaquopy provides the Python library using a Gradle plugin, rather than requiring a project to source a pre-compiled support package.
- We have addressed an issue with the way Briefcase handles requirements references when they are relative file references. Although this was added to allow rapid debugging of changes to Toga on Android, it also affected the recently added Linuxdeploy backend.
- We've added an "open" subcommand to Briefcase, making it easier to open project-based templates in their respective IDEs.
- We've improved Android log handling, ensuring that more app-specific logs are displayed to the user.
- We've dramatically improved our CI testing of Briefcase. We now build an app on every platform we support, using every backend.
- We've addressed a number of bugs in the Pack layout algorithm.
- We've continued the process of making Toga widget APIs naming internally consistent.
- We've fixed some canvas bugs that were causing problems on Windowns with our native Matplotlib rendering backend, toga-chart.
- We've addressed some bugs associated with changing the main content of a Toga window.
- We added an initial implementation of a DetailedList widget on Windows.
What's next?
In September, we hope to wrap up the work on binary dependencies. On Android, this is mostly a matter of completing the work to support multiple Python versions. On iOS, we need to develop the tools to manage the compilation of third-party binary libraries. We have an initial proof of concept of this; we need to finish these tools, and develop the infrastructure to distribute. We're also hoping to start conversations with the Anaconda team about Conda integration with Briefcase, and start improving GUI testing in Toga.
July 2022 Status Update
Another month of important updates to the BeeWare project! This month, our focus was on packaging improvements on Linux and Windows.
In addition the technical progress, we welcomed Malcolm Smith (@mhsmith on Github) to the Anaconda BeeWare team! Malcolm brings his considerable experience developing and maintaining Chaquopy, a set of tools and libraries for building Android applications that use Python. Chaquopy tackles the "Python on Android" problem from the perspective of adding Python to an existing Android Studio project rather than writing apps entirely with Python; as a result, it features much tighter integration with native Android Studio tooling. One particularly interesting feature of Chaquopy is that it supports binary dependencies on Android - a key feature that BeeWare's Android tooling currently lacks. Historically, Chaquopy was a closed source tool with licenses available for open source projects; however, as a result of joining Anaconda, Malcolm has released Chaquopy as an Open Source project!
What we've done
During July:
- We added support for Linuxdeploy plugins for Linux AppImages. These plugins are an important part of making AppImages self-contained, as many libraries need to provide the Linuxdeploy tooling hints to help find all the resources needed at runtime. Adding support for these plugins should address most of the issues we've been seeing with moving AppImages between Linux versions.
- We added a Flatpak backend for Linux packaging. Flatpak is an alternative packaging format to AppImage which has the support of a number of large Linux distributions. For now, AppImage continues to be the default Linux packaging format, but we may change this default in the future.
- We improved the packaging of Windows apps by adding a stub binary. We're still hunting some bugs in this stub binary, but when those are resolved, the Windows apps generated by Briefcase will present to the operating system with consistent app naming and icons. The stub binary also enables us to catch runtime problems with the app in a way that makes diagnosing app problems much easier.
- We presented a webinar about native application development. If you weren't able to attend the webinar live, a recording is available; register here to watch.
- We got an initial proof of concept of a Toga app running on Android using Chaquopy as a base, accessing a binary library (matplotlib), deployed with Briefcase. Over the coming weeks, we're hoping this proof of concept will evolve into major improvement to BeeWare's Android tooling.
- We modified Briefcase to use OS-appropriate directories for caching. While the ~/.briefcase location has served us well, it isn't a location that adheres to platform-native conventions for storing app resources, so we've moved Briefcase's cache of downloads, tools and templates to a platform-appropriate location.
- We addressed an issue with the current working directory leaking into the runtime path. This was especially common with Linux apps, but it could potentially be an issue with other platforms as well.
- We've started the process of making Toga widget APIs internally consistent. While we've aimed to keep naming of widgets' attributes and handlers consistent, there are a couple of annoying inconsistencies. We've started the process of cleaning up these inconsistencies.
- We've started looking into binary dependencies on iOS apps.
What's next?
In August, our major focus will be binary dependencies on mobile. This is one of the biggest missing pieces of the BeeWare story at present. Being able to utilize Chaquopy as a base gives us a huge head start on Android, so it seems likely we'll have a working solution for Android in the near future. However, on iOS, we're still at the "we don't know what we don't know" stage. It's unclear if we'll have a fully working solution by the end of the month, but we should have at least some progress to report.
June 2022 Status Update
It's been another busy month at the hive! This month, we've had a couple of high profile improvements, and some slow progress towards longer term goals. These improvements have been primarily in Briefcase and app packaging, but there have been some minor Toga improvements as well.
What we've done
During June:
- We significantly improved the UI of Briefcase by adding Rich to implement terminal output. This allows us to have vastly improved progress bars, as well as the ability to save a full debug log when a problem occurs. These improvements (and several others) are the result of contributions from community member Russell Martin - thanks Russell!
- We added support for macOS app notarization.
- We improved the way the Briefcase manages Android system images. This corrects a bug we were seeing that was causing the Android emulator to be installed twice, and could lead to situations where emulators could be discovered, but not started.
- We've explored the source of the Linux packaging problems we've been seeing. We've documented a workaround for one of those problems; and we're working on 2 fixes (supporting Linuxdeploy plugins and adding Flatpak support) that will hopefully address the remaining Linux packaging issues.
- We've improved Windows MSI installers so that it will use the right Program Files directory, as well as providing an install-time option for system vs user installs of a package.
- We made some small improvements that improve support for non-ASCII project names and descriptions.
- We've made some more improvements to Toga Webviews, ensuring that the current URL is accessible through the Python API.
- We've addressed a long standing bug that would cause Toga apps to be unable to find their local resource folders, depending on how the app was invoked. More importantly, this change has been accompanied by additions to the test suite. This is the start of being able to test runtime behavior of apps, which will be an increasinging important feature moving forward.
- We've added a stack trace dialog for Windows. A version of this dialog will be integrated into future improvements in Windows packaging.
- Finally, we published our Q3 roadmap.
What's next?
In July, we'll be focussing on:
- Completing the work on Linux and Windows application packaging.
- Presenting a webinar about native application development. If you'd like to attend, registration for this webinar is open
- Starting investigations into binary module support on mobile platforms.
We're also delighted to announce that we've successfully completed our hiring process; so, in 2 weeks, there will be another Bee in the hive! I'm very excited about the possibilities that this additional team member will afford. Look out for a welcome announcement in the coming weeks.
2022Q3 Roadmap
One of the benefits of having the support of an organisation like Anaconda is that we are now in a position to make more concrete commitments on project goals, and the likely timeframe when key features will be delivered. To that end, in addition to monthly status updates, we're going to start publishing regular updates on our project roadmap so the community has a better idea what to expect over the coming months.
Project estimation is difficult at the best of times; and project estimation on a FLOSS project is especially complex. In addition to any work that we are planning internally as a team, we need to manage contributions from the community. If a community member contributes a piece of work that isn't on our roadmap, we need to review that work. As a result, our ability to deliver on our own roadmap is dependent on how much is contributed by the community - something that is very hard to predict ahead of time.
In addition, several of the topics listed on this roadmap are complex, foundational tasks, and potentially contain unknown complications. As with any complex work, it can be difficult to predict ahead of time how long that work will take.
As a result, this roadmap should be read as a guide to what we aim to focus on over the coming quarter, rather than a hard commitment of features that will be made available on a specific deadline.
Q3 priorities
During Q3, we're planning to work on:
Linux packaging
Breifcase's AppImage packaging backend has had a cluster of issues reported related to inclusion of third party libraries, and the post-processing that needs to be performed on an AppImage. We've already started work on this during Q2; early in Q3, you should expect to see this work wrapped up. This will include:
- Adding support for linuxdeploy plugins for our existing AppImage backend; and
- Adding a Flatpak backend.
Windows packaging
Briefcase has an MSI packaging backend for Windows; and while this backend works, it doesn't deliver an end-user experience that is as polished as we'd like. We should be starting initial work on Windows packaging in the coming week, with the complete solution completed in early Q3. This will include:
- Adding a stub binary for Windows apps, so that Briefcase apps present to the task manager with the apps name and icon, rather than "pythonw";
- Improving log capture on Windows, so that it is easier to diagnose when problems occur; and
- Fixing some issues with the way the site is defined and used. This may also address some issues we've seen with binary modules in some Windows apps.
Binary modules
The biggest caveat on BeeWare's mobile app story has always been that we don't provide support for binary packages on mobile platforms. In Q3, we're going to start addressing this limitation.
There are already tools like crossenv that can simplify this process, and we've got a PR in our review queue that provides a proof-of-concept implementation of Android binary packaging. We need to take these tools, plus other tools in the Python ecosystem, and extend Briefcase so that we can present a clear development story for mobile developers that want to use modules with binary components.
On iOS, there is an additional complication - we need to resolve how to manage dynamic loading of modules. BeeWare's iOS support currently relies on static compilation of all binary modules. This was originally done because prior to iOS8, dynamic linking was prohibited on iOS. The days of iOS8 are now long behind us; but the Apple support package has retained this old approach, mostly because it wasn't clear what approach Apple would allow through the App Store review process. We will be making changes to Briefcase's iOS support to accomodate dynamic module loading, and then extending that support to include user-provided binary modules.
Testing (especially GUI testing)
Without testing, it's very difficult to ensure the stability and reliability of code in the long term. Unfortunately, many of the areas where BeeWare operates are inherently difficult to test. Visual GUI tests are notoriously difficult to perform; packaging tests require the ability to actually install and run finished applications; testing mobile applications requires access to mobile device hardware in a CI setup. As a project, we've avoided writing tests for many of our core features due to these complexities - and we've paid the price for this in regressions. Now that we have more time, we can focus on setting up the frameworks to let us perform this testing.
This includes adding:
- Support package tests, ensuring that the packages include valid libraries;
- App template tests, ensuring that Briefcase is able to generate valid and installable applications; and
- GUI tests, ensuring that Toga generates apps that look and behave the way they should.
Testing is an ongoing process, so it's highly unlikely we'll have "100% test coverage" by the end of the quarter. The goal will be to add the frameworks that allow us to add tests at all, so that we can retrofit tests onto existing code, and add tests on all new code going forward. Ideally, these tests will be fully automated tests running in CI. However, we may accept semi-automated or manual tests as a short term of interim measure, especially if those tests lay the groundwork for fully automated tests in the future.
Conda integration
Briefcase is currently uses PyPI and pip as source of Python packages. However, PyPI and pip isn't the only packaging solution in the Python ecosystem - another commonly used packaging source is Conda.
At this point, I must recognise the elephant in the room - Anaconda, the company that produced and maintains Conda, is my employer. However, I must stress that the fact this is on our roadmap is not as a result of a directive from my employer. The original proposal for this idea predates even my earliest hiring discussions with Anaconda. It is being pursued now for three reasons:
- The key value of Conda is that it simplifies the process of delivering complex binary packages, and ensuring that the binary dependencies for those packages are isolated from any packages that might be present on the user's computer. This core problem statement shares a lot of similarities with the core problem associated with application packaging. Conda also has much more nuanced representation of hardware platforms, alternatives for binary artefacts, and options for custom repositories. There is a possibility that using Conda as a packaging base may simplify some of the problems we've seen with distributing binary dependencies.
- There is a significant portion of the Python community that have commercial needs that requires the use of the managed repositories provided by Anaconda, or are simply more familiar with Conda tooling than PyPI/pip.
- As a result of having 1 (soon to be 2) full time engineers, we now have the resources to explore complex changes that might not have otherwise been explored. We are exploring this idea because we now have the resources to do so; we were not provided those resources on the condition that we explore this idea.
We don't anticipate dropping support for pip, or for PyPI as a package source. We're looking at extending Briefcase to provide another commonly used and requested option for packaging, rather than replacing PyPI/pip.
Longer term goals
Looking beyond Q3, the big items on the horizon are:
Upstreaming
The Apple and Android support packages maintained by BeeWare include a number of patches against the core CPython. We want to begin the process of getting these modifications into the CPython codebase.
There have been two major impediments to this in the past:
- Development resources When we were a volunteer driven project, we didn't have the resources to engage meaningfully with the CPython core team to shepherd our patches into the CPython codebase.
- Testing Supported platforms in the CPython codebase need to have CI solutions.
We now have resources, so we can dedicate the time needed to upstream patches; and once we've completed our testing work in Q3, we'll be in a position to recommend CI configurations to the CPython core team. That means the work of upstreaming patches and making iOS and Android fully supported platforms can begin.
Toga 1.0
The broad goal for "Toga 1.0" could be considered "Tkinter, but with some additional bells to show off". We want to deliver a core widget set of the "essentials", plus some nice features to show off, like WebViews and camera or GPS integration. The exact set of features for Toga 1.0 is still a subject for discussion; I'd expect to see a more concrete feature list and plan emerge in Q4.
Summary
As with most of the work that has been done in Q2, the foundational work we're planning for Q3 is not as glamorous as adding high profile GUI widgets, but it's all necessary work if we're going to provide a stable and reliable development story. Once this foundation is established, we'll be in a much better position to rapidly develop new features that are visible - and to ensure that those features are stable and reliable as BeeWare develops.
May 2022 Status Update
It's been another busy month at the hive! The progress we made in May isn't as obvious as last month - but the changes that have been made are an essential part of ensuring we have a stable foundation for future work on the BeeWare ecosystem.
What we've done
During May:
- We updated the Apple support packages. These updates aren't just a version bump - they also added support that enables the iOS emulator to run natively on M1 devices. This required a major restructure of the way the support packages are bundled, using Apple's new XCframework library packaging format.
- We addressed an issue with the use of ctypes on physical iOS devices. ctypes is a core part of the infrastructure we use to integrate with the iOS system libraries, so resolving this bug was critical for distributing apps. The fix was deep in the core of libFFI, and has been submitted for integration into the upstream project.
- We released an Android support package for Python 3.10, as well as updated support packages for Python 3.7, 3.8, and 3.9. These support packages dramatically are now dramatically smaller, and result in smaller end-user applications, thanks to some compiler flag optimisations suggested by the community.
- We began the process of adding support for Python 3.11 to the Apple and Android support packages. This is the first time BeeWare has been able to contribute prerelease testing to the CPython project. In the process, we discovered a small backwards incompatibility. Baring major incidents, we should be in a position to release BeeWare's support for Python 3.11 on the same day as the official 3.11.0 is finalised.
- We developed an acceptance test suite for the Apple and Android support packages. This test suite tests for features in the Python core that have a history of being broken (such as modules in the standard library that have binary components, but have been incorrectly compiled or inadvertently excluded from the build). This test suite isn't yet formally integrated into the build process, but the hope is that it will be in the near future.
- We modified Briefcase so that it is able to run the Android emulator on M1 hardware. This required a major rebuild of the way the Android SDK tooling is managed.
- We made further improvements to the log capture tools on macOS. This ensures that the first few seconds of application logs are reliably captured, and the log stream is terminated when the app exits.
- We started the introduction of Black as a part of BeeWare's development tooling. BeeWare's repositories have always enforced code style guidelines, and those code styles are very closely aligned with Black's choices. With Black formally moving out of Beta status at the start of this year, the time has come to start adopting Black as a formal project requirement. For now, Briefcase is the only project that has formally adopted Black, but you can expect this to continue to roll out over the other projects in BeeWare over the coming months.
- We resolved some minor bugs that were present in the 0.3.0dev33 release of Toga. Most important of these was a bug that prevented the BeeWare tutorial running on iOS devices.
Many of these changes either began as, or are entirely the work of community members. There are also plenty of smaller PRs, bug fixes, and bug reports that are just as important, but are too numerous to list. A huge thanks to everyone who has contributed to BeeWare during May.
What's next?
In June, we'll be focussing on:
- Publishing our roadmap for Q3 and beyond! Now that we have reliable development resources, we're in a position to make public commitments on the future direction of the project.
- Completing the hiring process for another engineer to work on BeeWare full time. This hiring process is underway, but it's not too late to apply if you're interested. Full details of the position can be found on Greenhouse.
- App Notarization on macOS. This is becoming an increasingly required part of macOS app publishing, as Apple cranks up their security requirements.
- Improving Windows packaging. This includes improving the way the running app presents to the Task Manager, improving log capture, and app signing.
- Improving Linux packaging. This includes addressing some issues with packaging binary dependencies.
Now Hiring!
BeeWare is now hiring!
The position is a full time, Mid- to Senior position. You will be working full time in the Open Source group at Anaconda, on the BeeWare suite of tools. Full details of the position can be found on Greenhouse.
The position calls for an unusual combination of skills. The ideal candidate would have experience building GUI applications (especially mobile) and Python skills. However, because of the existing state of the Python ecosystem, most Python developers don't have GUI development experience, and most GUI developers don't have extensive Python experience. For that reason, if the position is interesting to you, but you don't have all the "must have" attributes - I would encourage you to apply anyway. A candidate with no GUI development experience will still considered, as long as they've got a demonstrated history of doing weird and wonderful things with Python. Similarly, a developer with deep GUI experience, but no Python experience, will also be considered.
The job location requirements are also unusual. The position is remote; the position requires that your working hours need to be compatible with UTC+8. This means candidates from Australia, South East and South Asia will be a natural fit. European candidates will need to be prepared for early morning starts. US/Canadian candidates will need to be prepared for evening work (very late evenings if you're in CST or EST timezones). Anaconda has the capacity to hire in the UK, Germany, India, Australia, US, and Canada. If you're not a resident of one of those countries, it may be possible to hire you, but it will likely require you to operate as a private contractor rather than a salaried employee.
I'm incredibly excited for what the future holds for BeeWare - if you'd like to come on this journey with me, please apply (and tell them Russell sent you)!