The Buzz

Q4 progress

Progress in Q4 was not as significant as we originally planned. A combination of conference travel and holiday leave meant the quarter was shorter than most; but even then, our goals for the quarter were extremely optimistic.

We were able to hit our goal for Web deployment. We've restored the Toga web backend to a working state, and we've added a Web deployment backend to Briefcase. There's still a lot of work required before Web deployment is a viable as a production solution, but the initial pieces are now in place.

We were also able to hit our goals on housekeeping. We've made a lot of improvements to the base infrastructure of repositories, enabling Black, pre-commit, and towncrier in Toga, and improving coverage reporting (including getting to 100% branch coverage in Briefcase's test suite).

We've made significant progress on improving the testing story for BeeWare apps. We've added the ability for Briefcase projects to run normal Python test suites inside a packaged app, and we've demonstrated that we can use this capability to run CI for iOS and Android apps (as well as desktop platforms). This feature will be included in the next Briefcase release.

Using this new testing capability, we've started building a GUI test suite for Toga. This is a critical part of ensuring the quality of Toga going forward; but getting this testing framework in place has revealed lots of complications and bugs. We're making progress, but there's still a lot of work to be done.

Our plans to release Toga v0.3 final have also been delayed. The process of adding SCM-based versioning to the repository (part of the housekeeping task) revealed a bug that prevented code running on Android; fixing that bug became a blocker on the release of Toga v0.3. We've now resolved that problem, so we should be able to release Toga v0.3 final as soon as the fix has been released into the BeeWare's Android tooling.

Unfortunately, that leaves a lot on the Q4 TODO list. We haven't made any changes to the example apps; Toga documentation is mostly unchanged; we haven't been able to do any substantial Toga planning; and we haven't been able to add any native cross-platform hardware support.

Q1 priorities

It's probably unsurprising that Q1 2023 will be picking up on items we didn't get around to in Q4 2022.

• Testing - Completing the Toga GUI testing suite will be our primary focus for the quarter.
• Planning - Once we've got the test harness in place, we'll be in a much better position to evaluate what work remains to get to v1.0.
• Documentation - The process of systematically building a test suite also gives us an opportunity to update the API documentation for each widget as we go. There's still lots of documentation required on top of the API guide, but complete API documentation will be a big start.
• Cross-platform hardware - Cross-platform support for device hardware (especially mobile) is one of the banner features that makes mobile device support worthwhile. Providing proof that this hardware support is easy to use will be a key demonstrator for BeeWare as a platform.

Longer term goals

Our longer term goals are largely unchanged from Q4 2022. Upstreaming, Toga 1.0, and a forge for mobile packages are all important goals for the BeeWare ecosystem. However, we don't expect to make any significant progress on these goals until we have solidified Toga's testing story.

Summary

The seeming lack of progress made in Q4 was definitely disappointing; but the work hasn't been wasted. Progress has been slow specifically because the problems we're trying to solve are hard, and reveal every gnarled edge case that platforms have to offer. We are making progress, though; and once we've got the Toga testing framework stabilized, we'll be in a much stronger position to make predictions for future work on Toga.

What we've done

During December:

• We continued to fine tune Briefcase's recently added test mode. This reflects issues that have been discovered as we continue to build Toga's GUI test suite.
• We updated Briefcase's Web backend to support changes made in the 2022.12.1 release of PyScript.
• We fixed a bug preventing the use of local file references in requirements when building Linux AppImages. As a side effect, this change also makes it much easier to get a debug session inside the Docker container used to build AppImages.
• We merged the first draft of the Toga GUI testbed. There is still a lot of work to be done on this testbed, but it provides a solid foundation on which we can build tests of Toga's cross-platform GUI behavior.
• We added support for mouse events on the Canvas on GTK.
• We corrected a bug on Windows where local date formats would cause the DateInput widget to crash.
• We fixed a bug in Rubicon that would cause background threads to stall on iOS.

What's next?

There won't be much more progress from the core team for the rest of this year. We'll still be around to handle critical problems, answer questions and do code reviews for contributors; but our response times might be a little slower than normal. We'll publish our Q1 2023 roadmap when we return in January - but we'll be largely picking up where this year has left off - improving the testing story for Toga.

Want to get involved?

Want to get involved? Here are some open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:

1. Modify the handling of app signing options
2. Filter out a message generated after Xcode updates
3. Add the ability to refresh the support package in a project
4. Provide protection against creating projects with "semi-reserved" names
5. Add the ability to configure the ABIs built by an Android project
6. Add support for the ANDROID_HOME environment variable
7. Purge cache folders when installing app code in Briefcase

Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.

What we've done

During November:

• We added a test mode to Briefcase. This is a huge step forward, as it means we are now able to run automated tests for iOS and Android, as well as validation tests for packaged app behaviors. As a proof of concept, we've already begun porting our support package validation app - this is an app that we've historically needed to run manually.
• We added the ability for an AppImage to customise the Dockerfile used to build the project. This was required to support the use of Rust-based dependencies (such as cryptography) in AppImages; however, the approach we've taken should allow for any other customisation that may be required, without the need to maintain a fork of the AppImage project template.
• We've added the ability for users to add customisations to the Android Gradle file for their projects. As with the Dockerfile changes, this allows for small customisations without the need to use a completely forked template.
• We improved the resilience of Briefcase on flaky network connections. Previously, an interrupted download of a support package could leave Briefcase in a broken state; this should no longer be possible.
• We've improved type hinting in the Briefcase codebase, especially relating to the ToolCache class.
• We've improved the handling of Android apps that fail immediately on app startup. Previously, if your app crashed due to something like a Python syntax error, it was possible for Briefcase to hang in a way that made it look like the issue was due to Briefcase being unable to start the app, rather than a user code problem.
• We've corrected a problem on Windows that was preventing app output from being captured in the Briefcase log.
• We completed the process of modernising the layout and tooling of the Toga repository, including the introduction of Black, Towncrier, setuptools_scm, and a large reorganisation of the code structure
• We've begun building the first automated test of Toga's GUI behavior. This is still in the early stages, but it will form the a core part of our testing infrastructure in the near future.
• We've added preliminary support for Python 3.12 to Briefcase, Toga and Rubicon ObjC.
• We've updated the codebase of Rubicon-ObjC to use Black.

What's next?

December will be a slow month due to the Christmas and New Year break; however, until then, we'll continue to focus will continue to be on testing. We'll continue to build out the testing tools that we've worked on this month, and start building tests using those tools. We also hope to begin work on a firm 1.0 roadmap for Toga.

Want to get involved?

Want to get involved? Here's 8 open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:

1. Modify the handling of app signing options
2. Filter out a message generated after Xcode updates
3. Add the ability to refresh the support package in a project
4. Provide protection against creating projects with "semi-reserved" names
5. Add the ability to configure the ABIs built by an Android project
6. Add support for the ANDROID_HOME environment variable
7. Add a command summary to briefcase --help
8. Add an option to display the platforms supported by a command

Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.

What we've done

During October:

• We restored the web backend for Toga! Web deployment has always been part of the BeeWare story, but we hit some technological challenges with the approach that we were taking in the Toga 0.2 branch. However, with the addition of WASM as an officially supported platform in CPython 3.11, and the release of PyScript, we have a new way of getting Python in the browser. We have been able to use these tools to build a much improved Toga web backend.
• We added support for deploying Briefcase apps as static web sites.
• We released official support for Python 3.11 on macOS, iOS and Linux - on the day that Python 3.11 was released!. Python 3.11 support for Android and Windows is ready to go, and should be published in a day or two.
• We modified Briefcase to write its logs into subdirectory, instead of littering them in the project directory.
• We presented a talk at DjangoCon US 2022. The video for this talk should be up in around a month. A major feature of this talk was a demo of an Electron-style "web site as an app", built entirely in Python. This demo (called Positron) has been included in the Toga repository as an example.
• We attended the DjangoCon US 2022 sprints, and handed out 4 challenge coins to new contributors!
• We added the ability to retrieve widgets by ID to the Toga API.
• We made a big change to the way that Toga backends are discovered. This new approach has two major advantages - firstly, it means third-parties can register their own backends; but more importantly, it means that code no longer needs to explicitly provide a backend. This simplifies the creation of test cases - but it also means that icons, images and fonts no longer need to be "late bound". This is an internal detail, but it significantly simplifies the code using those data types.
• We added the ability to create Image objects from raw data, rather than an image file.
• We started the process of modernising the layout and tooling of the Toga repository. Some of these changes have already landed; more will come in the coming days. This ensures that we have a stable foundation for future work on Toga.

What's next?

During November, we continuing to focus on Toga, with a particular focus on testing. A particular focus will be working out how to test graphical features of Toga - an area that has historically been managed entirely with manual testing. Having strong automated testing of graphical features will be a key milestone in ensuring Toga remains stable in the long term.

Want to get involved?

Want to get involved? Here's 8 open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:

1. Improve protection against corrupted or incomplete downloads
2. Modify the handling of app signing options
3. Filter out a message generated after Xcode updates
4. Add the ability to refresh the support package in a project
5. Provide protection against creating projects with "semi-reserved" names
6. Add the ability to configure the ABIs built by an Android project
7. Add support for the ANDROID_HOME environment variable
8. Silence an warning that is displayed when an Android emulator has no skin defined

Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.

Q3 progress

We were able to hit our biggest Q3 goals:

• Linux packaging - We've investigated and addressed all the known issues with Linux packaging. There's one pending enhancement that would expand the range of third-party binary packages that can be included in Briefcase projects.
• Windows packaging - We've introduced a stub binary for Windows apps (as well as a Visual Studio target for Windows projects). This allowed us to significantly improve log capture, so when a Windows app crashes it's a lot easier to obtain diagnostic information. This change also corrected most of the issues with the way Windows apps were using the site module; however, there is still one issue related to the handling of packages that have complex post-installation processes.
• Binary modules - By far the biggest achievement of Q3. With the release of Briefcase 0.3.10, we now support binary modules on iOS and Android. We have also published 40 of the most popular binary modules, compiled for iOS and Android.

However, we didn't hit all our goals:

• Testing - Testing support has improved substantially. We have a much more thorough CI configuration on Briefcase that verifies that templates generate valid projects. We also have a testbed project that we can use to check that support packages are behaving correctly. However, we are not yet able to run this test project in CI, so we don't have automated validation of PRs against the support packages. We also made very little progress in GUI testing.
• Conda integration - We began initial discussions around Conda integration, on two fronts: firstly, to investigate the possibility of using Conda as a base for package management in apps; and secondly, about using conda-forge as a framework for managing community contributions. Both of these conversations are ongoing; it will likely take some time before these efforts bear fruit.

Q4 priorities

In Q4, we're planning to focus on:

Longer term goals

Looking into next year, the big items on the horizon are:

Summary

For the last 6 months, the focus of BeeWare has been on Briefcase and the support packages - ensuring that we have a feature complete, robust set of tools for deploying apps on any platform. In Q4, our focus is shifting to the tools we need to build apps on those platforms, and ensuring we have a stable foundation for future development. It's taken a long time for BeeWare to get this far - but we're getting very close to unlocking the remarkable potential of the Python ecosystem on mobile devices.

What we've done

During September:

• We added support for binary packages on mobile platforms! There's more detail related to this than can fit in a bullet point; more details below.
• We added the ability to remove files from a templated app. This can be used to trim down a project, removing parts of the Python interpreter that you don't need (such as standard library modules that you don't need)
• We've changed the way we use templates, switching to using branches based on the Briefcase release, rather than the Python release. This enables us to make development changes to templates without affecting the published stable release.
• We reworked the way Briefcase manages integrations with the tools needed to build apps. This was a mammoth project, but as a result the process of verifying tools is faster, and a number of bugs related to having stale builds have been resolved.
• We've made a number of improvements to console handling of log messages, especially on Windows
• We improved error handling in Toga when TLS1.2 or TLS1.3 isn't supported by Windows.
• We started conversations with the teams at Anaconda responsible for conda about using Conda as a base for Briefcase apps, and on using Conde-forge style infrastructure to support binary packaging. Nothing concrete has come from these conversations yet; these conversations are ongoing.

What's next?

For the last few months, the focus of the BeeWare team has been on Briefcase and the support packages. In October (and the rest of Q4), we'll be shifting our focus to Toga. The initial focus will be on improving the foundation of the project - testing and build infrastructure; but once those pieces are in place, we'll start adding new features.

We'll also be at DjangoCon US in San Diego - and there will be some web-related updates related to the presentation that will be given at that conference. We hope we'll see you there!

Want to get involved?

Want to get involved? Here's 10 open issues that would be a great place to get started with contributing to a BeeWare project. They're all relatively minor changes, but would provide a big improvement to the lives of BeeWare users:

1. Write Briefcase logs into a subdirectory, instead of the project folder
2. Improve protection against corrupted or incomplete downloads
3. Modify the handling of app signing options
4. Modify the Briefcase new project wizard to show which repository and branch it's using
5. Filter out a message generated after Xcode updates
6. Add the ability to refresh the support package in a project
7. Provide protection against creating projects with "semi-reserved" names
8. Add the ability to configure the ABIs built by an Android project
9. Add support for the ANDROID_HOME environment variable
10. Silence an warning that is displayed when an Android emulator has no skin defined

Pick one of these tickets, drop a comment on the ticket to let others know you're looking at it, and try your hand at a PR! We have a guide on setting up a Briefcase development environment; but if you need any additional assistance or guidance, you can ask on the ticket, or join us on the BeeWare Discord server.

What we've done

During August:

• We have continued to work on the changes needed to migrate Android support to Chaquopy. This has included some major changes to the Android Gradle template, and developing a Rubicon Java compatibility layer. Eventually, Toga will migrate to use the Chaquopy APIs directly, but in the interim, the Rubicon APIs will continue to work.
• We have started updating Chaquopy to support multiple Python versions. Historically, Chaquopy has only supported a single Python3 version (3.8); for compatibility with Briefcase, we are adding support for Python 3.9 and 3.10, as well as preparing the ground for a 3.11 release when Python 3.11 is finalised.
• We have modified the Python Apple support packages to provide the binary portions of the Python standard library as dynamically loaded binary modules, and modified the iOS and macOS templates to use and sign these binary modules. These changes haven't landed yet, but they have been tested on simulators and physical devices. Along the way, we've also updated the support packages to use OpenSSL 3.
• We have modified Briefcase to make the Python support package optional, at the discretion of the project template. This was needed because Chaquopy provides the Python library using a Gradle plugin, rather than requiring a project to source a pre-compiled support package.
• We have addressed an issue with the way Briefcase handles requirements references when they are relative file references. Although this was added to allow rapid debugging of changes to Toga on Android, it also affected the recently added Linuxdeploy backend.
• We've added an "open" subcommand to Briefcase, making it easier to open project-based templates in their respective IDEs.
• We've improved Android log handling, ensuring that more app-specific logs are displayed to the user.
• We've dramatically improved our CI testing of Briefcase. We now build an app on every platform we support, using every backend.
• We've addressed a number of bugs in the Pack layout algorithm.
• We've continued the process of making Toga widget APIs naming internally consistent.
• We've fixed some canvas bugs that were causing problems on Windowns with our native Matplotlib rendering backend, toga-chart.
• We've addressed some bugs associated with changing the main content of a Toga window.
• We added an initial implementation of a DetailedList widget on Windows.

What's next?

In September, we hope to wrap up the work on binary dependencies. On Android, this is mostly a matter of completing the work to support multiple Python versions. On iOS, we need to develop the tools to manage the compilation of third-party binary libraries. We have an initial proof of concept of this; we need to finish these tools, and develop the infrastructure to distribute. We're also hoping to start conversations with the Anaconda team about Conda integration with Briefcase, and start improving GUI testing in Toga.

What we've done

During July:

• We added support for Linuxdeploy plugins for Linux AppImages. These plugins are an important part of making AppImages self-contained, as many libraries need to provide the Linuxdeploy tooling hints to help find all the resources needed at runtime. Adding support for these plugins should address most of the issues we've been seeing with moving AppImages between Linux versions.
• We added a Flatpak backend for Linux packaging. Flatpak is an alternative packaging format to AppImage which has the support of a number of large Linux distributions. For now, AppImage continues to be the default Linux packaging format, but we may change this default in the future.
• We improved the packaging of Windows apps by adding a stub binary. We're still hunting some bugs in this stub binary, but when those are resolved, the Windows apps generated by Briefcase will present to the operating system with consistent app naming and icons. The stub binary also enables us to catch runtime problems with the app in a way that makes diagnosing app problems much easier.
• We presented a webinar about native application development. If you weren't able to attend the webinar live, a recording is available; register here to watch.
• We got an initial proof of concept of a Toga app running on Android using Chaquopy as a base, accessing a binary library (matplotlib), deployed with Briefcase. Over the coming weeks, we're hoping this proof of concept will evolve into major improvement to BeeWare's Android tooling.
• We modified Briefcase to use OS-appropriate directories for caching. While the ~/.briefcase location has served us well, it isn't a location that adheres to platform-native conventions for storing app resources, so we've moved Briefcase's cache of downloads, tools and templates to a platform-appropriate location.
• We addressed an issue with the current working directory leaking into the runtime path. This was especially common with Linux apps, but it could potentially be an issue with other platforms as well.
• We've started the process of making Toga widget APIs internally consistent. While we've aimed to keep naming of widgets' attributes and handlers consistent, there are a couple of annoying inconsistencies. We've started the process of cleaning up these inconsistencies.
• We've started looking into binary dependencies on iOS apps.

What's next?

In August, our major focus will be binary dependencies on mobile. This is one of the biggest missing pieces of the BeeWare story at present. Being able to utilize Chaquopy as a base gives us a huge head start on Android, so it seems likely we'll have a working solution for Android in the near future. However, on iOS, we're still at the "we don't know what we don't know" stage. It's unclear if we'll have a fully working solution by the end of the month, but we should have at least some progress to report.

What we've done

During June:

• We significantly improved the UI of Briefcase by adding Rich to implement terminal output. This allows us to have vastly improved progress bars, as well as the ability to save a full debug log when a problem occurs. These improvements (and several others) are the result of contributions from community member Russell Martin - thanks Russell!
• We added support for macOS app notarization.
• We improved the way the Briefcase manages Android system images. This corrects a bug we were seeing that was causing the Android emulator to be installed twice, and could lead to situations where emulators could be discovered, but not started.
• We've explored the source of the Linux packaging problems we've been seeing. We've documented a workaround for one of those problems; and we're working on 2 fixes (supporting Linuxdeploy plugins and adding Flatpak support) that will hopefully address the remaining Linux packaging issues.
• We've improved Windows MSI installers so that it will use the right Program Files directory, as well as providing an install-time option for system vs user installs of a package.
• We made some small improvements that improve support for non-ASCII project names and descriptions.
• We've made some more improvements to Toga Webviews, ensuring that the current URL is accessible through the Python API.
• We've addressed a long standing bug that would cause Toga apps to be unable to find their local resource folders, depending on how the app was invoked. More importantly, this change has been accompanied by additions to the test suite. This is the start of being able to test runtime behavior of apps, which will be an increasinging important feature moving forward.
• We've added a stack trace dialog for Windows. A version of this dialog will be integrated into future improvements in Windows packaging.
• Finally, we published our Q3 roadmap.

What's next?

In July, we'll be focussing on:

• Completing the work on Linux and Windows application packaging.
• Presenting a webinar about native application development. If you'd like to attend, registration for this webinar is open
• Starting investigations into binary module support on mobile platforms.

We're also delighted to announce that we've successfully completed our hiring process; so, in 2 weeks, there will be another Bee in the hive! I'm very excited about the possibilities that this additional team member will afford. Look out for a welcome announcement in the coming weeks.

Q3 priorities

During Q3, we're planning to work on:

Longer term goals

Looking beyond Q3, the big items on the horizon are:

Summary

As with most of the work that has been done in Q2, the foundational work we're planning for Q3 is not as glamorous as adding high profile GUI widgets, but it's all necessary work if we're going to provide a stable and reliable development story. Once this foundation is established, we'll be in a much better position to rapidly develop new features that are visible - and to ensure that those features are stable and reliable as BeeWare develops.