Managing plugins in Logic Pro X with Python (part 3)

11/12/2019

I spend way too much time making sure I add every new plugin I buy to a plugin manager category, else it gets forgotten and lost in the depths of the plugins by manufacturer section - so I decided to see if I could improve this a little with python.

Now that the previous posts have covered how Logic Pro's plugin manager stores data and how to scan for plugins, we can bring this together with a Python script that will be able to inspect that data.

Intro

This post is part of a series which will end with some C++ and Python code that can scan for plugins and alert us if a plugin hasn't been added to a category. In the process we'll cover the following: I assume that you're already familiar with the plugin manager and how to use it to create categories. If not, I recommend reading this article (link) for an introduction.

This post will be the last in the series, and will focus on using Python to parse the plugin manager data, and scheduling the script to run with launchctl.

If you have any issues following the steps in this post, you can refer to the complete implementation at this repo (link), which also includes some additional features and optimisations.

Python prerequisites

Luckily there aren't any third party packages required to make this work, everything needed is included with a standard Python 3 installation. In this case we're using Python 3.6, but other versions should also work with some tweaking.

Calling our PluginScanner from Python

In the previous post we created a PluginScanner class, and gave it a C interface that can be called from Python. Now we need to write the Python side of this interface.

When the PluginScanner project is compiled into a dynamic library, it produces a file inside the project's directory under Builds/MacOSX/build/Debug/JUCEBinding.dylib (or a <some other name>.dylib depending what you called your project). This file contains the compiled PluginScanner code that we need to call from Python.

Now we create a file called JUCEWrapper.py, and add the following lines:

"""
Python wrapper for the JUCE based dll.
"""

from ctypes import cdll, c_char_p, c_void_p, c_int
import os
import sys


These lines do the following:

A frustrating problem I ran into while writing this script was the overwhelming amount of output that our PluginScanner printed to the screen. This made it very hard to debug issues, so let's address this problem now.

The unnecessary output isn't generated by code that we wrote, but code that is inside the JUCE functions that we call. This makes it difficult to suppress the output from within our C++ code, so instead we'll do it using Python.

Everywhere that we call our PluginScanner C interface from Python, we'll wrap it in a "with" block. This would look something like the following:

with SuppressStream(sys.stderr):
    with SuppressStream(sys.stdout):
        // Do something that calls PluginScanner


Notice that we actually use two nested "with" blocks - one for each output stream that we want to suppress. This is required since parts of the excessive output that we're trying to remove appear in both stderr and stdout.

So what is the SuppressStream object that we're creating here? It's a fairly simple object which takes a stream as a parameter in its constructor, then it suppresses the stream when we enter the with block, and removes the suppression when the block is exited.

Using a with block in this way is a neat way to achieve the desired outcome, without having to do something like rely on calling methods of SuppressStream to tell it when to start and stop suppressing output. This would rely on the developer to remember when to call these methods and may be prone to errors.

The SuppressStream code below should be copied into your file directly below the import statements:

class SuppressStream():

    def __init__(self, stream=sys.stderr):
        self.orig_stream_fileno = stream.fileno()

    def __enter__(self):
        self.orig_stream_dup = os.dup(self.orig_stream_fileno)
        self.devnull = open(os.devnull, 'w')
        os.dup2(self.devnull.fileno(), self.orig_stream_fileno)

    def __exit__(self, type, value, traceback):
        os.close(self.orig_stream_fileno)
        os.dup2(self.orig_stream_dup, self.orig_stream_fileno)
        os.close(self.orig_stream_dup)
        self.devnull.close()


As you can see it has 3 parts:

Now that is done, we can look at our interface to PluginScanner. Copy the following directly below SuppressStream:

class PluginScanner:
    def __init__(self):
        SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))
        LIB_PATH = os.path.join(SCRIPT_DIR, "JUCEBinding/Builds/MacOSX/build/Debug/JUCEBinding.dylib")

        self.lib = cdll.LoadLibrary(LIB_PATH)
        self.lib.PluginScanner_new.argtypes = []
        self.lib.PluginScanner_new.restype = c_void_p

        with SuppressStream():
            with SuppressStream(sys.stdout):
                self.obj = self.lib.PluginScanner_new()

    def getFileOrIdentifier(self, index):
        self.lib.PluginScanner_getFileOrIdentifier.argtypes = [c_void_p, c_int]
        self.lib.PluginScanner_getFileOrIdentifier.restype = c_char_p

        with SuppressStream():
            with SuppressStream(sys.stdout):
                retVal = self.lib.PluginScanner_getFileOrIdentifier(self.obj, index).decode("utf-8")

        return retVal

    def getManufacturer(self, index):
        self.lib.PluginScanner_getManufacturer.argtypes = [c_void_p, c_int]
        self.lib.PluginScanner_getManufacturer.restype = c_char_p

        with SuppressStream():
            with SuppressStream(sys.stdout):
                retVal = self.lib.PluginScanner_getManufacturer(self.obj, index).decode("utf-8")

        return retVal

    def getName(self, index):
        self.lib.PluginScanner_getName.argtypes = [c_void_p, c_int]
        self.lib.PluginScanner_getName.restype = c_char_p

        with SuppressStream():
            with SuppressStream(sys.stdout):
                retVal = self.lib.PluginScanner_getName(self.obj, index).decode("utf-8")

        return retVal

    def getNumPlugins(self):
        self.lib.PluginScanner_getNumPlugins.argtypes = [c_void_p]
        self.lib.PluginScanner_getNumPlugins.restype = c_int

        with SuppressStream():
            with SuppressStream(sys.stdout):
                retVal = self.lib.PluginScanner_getNumPlugins(self.obj)

        return retVal


This looks a little complicated at first, so lets go through each part. We now have Python class called PluginScanner, which has all the same methods as our C++ PluginScanner, except each of the Python class's methods actually just call the C++ class's methods through its C interface. At This point Python has no knowledge of the C++ code, it just sees the C interface that we wrote for it.

Let's look at the constructor of our Python PluginScanner that we just wrote:

def __init__(self):
    SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))
    LIB_PATH = os.path.join(SCRIPT_DIR, "JUCEBinding/Builds/MacOSX/build/Debug/JUCEBinding.dylib")

    self.lib = cdll.LoadLibrary(LIB_PATH)
    self.lib.PluginScanner_new.argtypes = []
    self.lib.PluginScanner_new.restype = c_void_p

    with SuppressStream():
        with SuppressStream(sys.stdout):
            self.obj = self.lib.PluginScanner_new()


The first thing it does it put together the path to the dylib containing our C++ PluginScanner. You may need to change the value of LIB_PATH if you put your script in a different place or named your JUCE project differently.

Then it uses cdll.LoadLibrary (from ctypes) to load the dylib file and store it in self.lib. Next is one of the more fiddly parts of the process - telling Python's ctypes package what types the functions in our C interface expect as input and return as output.

From the previous post, our PluginScanner_new function takes no arguments, and returns a pointer to the C++ PluginScanner object that it constructs. In the constructor we tell ctypes this by writing:
    self.lib.PluginScanner_new.argtypes = []
    self.lib.PluginScanner_new.restype = c_void_p
c_void_p is one of the types we imported from ctypes, and represents a pointer to any object.

Finally the last part of our constructor calls the PluginScanner_new function and stores the output in self.obj, as we'll be passing this object to our other C functions (such as PluginScanner_getManufacturer) when we call them later.

Now let's look at one of the methods of our Python PluginScanner class:

def getManufacturer(self, index):
    self.lib.PluginScanner_getManufacturer.argtypes = [c_void_p, c_int]
    self.lib.PluginScanner_getManufacturer.restype = c_char_p

    with SuppressStream():
        with SuppressStream(sys.stdout):
            retVal = self.lib.PluginScanner_getManufacturer(self.obj, index).decode("utf-8")

    return retVal


As in the constructor, we tell ctypes what our function's inputs and outputs are by setting argtypes and restype, only this time the function is PluginScanner_getManufacturer. Then we call PluginScanner_getManufacturer, passing it the PluginScanner pointer stored in self.obj and the index of the plugin to lookup (as passed to the Python getManufacturer method itself).

This is repeated for each method, the only differences being the data types specified in argtypes and restype. That's it for the wrapper code, now we can finally move on to the main script.

Scripting

Now we'll write our script which puts everything together. My aim in writing this script is to be notified whenever my plugins aren't arranged into categories quite how I like them, which is to simply have each plugin be a member of exactly one category. With a few modifications you could make your script work a little differently to suit however your like your plugins to be organised.

To achieve this the script will do the following: We start by creating a new file called main.py and add the following:

"""
Checks if all detected plugins are in a single category.
"""

import binascii
import json
import os
import pickle
import sys
import xml.etree.ElementTree as ET

from JUCEWrapper import PluginScanner

ROOT_DIR = os.path.dirname(os.path.realpath(__file__))


As before we're importing some useful packages, but also our Python PluginScanner class. Now we need to add a few helper classes and functions:

class TagsetError(Exception):
    """
    Raised in the event that a tagset can't be parsed.
    """
    pass

class ScannedPlugin:
    """
    Stores the details for a single plugin.
    """
    def __init__(self, name, manufacturer, identifier):
        self._name = name
        self._manufacturer = manufacturer
        self._identifier = identifier

    def getName(self):
        return self._name

    def getManufacturer(self):
        return self._manufacturer

    def getIdentifier(self):
        return self._identifier

def postNotification(title, content):
    os.system(f"osascript -e 'display notification \"{content}\" with title \"{title}\"'")

def suppressStream(stream):
    orig_stream_fileno = stream.fileno()
    orig_stream_dup = os.dup(orig_stream_fileno)
    devnull = open(os.devnull, 'w')
    os.dup2(devnull.fileno(), orig_stream_fileno)

def codeToHex(code):
    """
    Converts a type code to its hex representation.
    """
    return binascii.hexlify(bytes(code, encoding="utf-8")).decode("utf-8")

def buildTagSetFileName(identifier):
    """
    Converts somthing like 'AudioUnit:Effects/aufx,clu2,SNSH'
    to something like 61756678-73686674-534e5348.tagset
    """

    typeCodes = identifier.split("/")[1].split(",")

    return codeToHex(typeCodes[0]) + "-" + codeToHex(typeCodes[1]) + "-" + codeToHex(typeCodes[2]) + ".tagset"


Let's go through what each of these classes and functions do: Now we add one final helper function. This one is important as it parses the tagset XML file and figures out what categories a plugin is a member of, and raises the TagsetError if parsing fails.

def getCategoriesFromTagset(tagsetPath):
    """
    Assumes the only dict tag is the one we are interested in.

    Raises TagsetError if any exceptions are raised when parsing.
    """
    try:

        tree = ET.parse(tagsetPath)
        root = tree.getroot()

        dictTag = root.find("dict")

        categories = []

        subDictTag = dictTag.find("dict")

        categoryTags = subDictTag.findall("key")

        categories = [categoryTag.text for categoryTag in categoryTags]

    except Exception as e:
        raise TagsetError(str(e))

    return categories


Now we can start the main part of our script by creating an instance of the PluginScanner that calls our C++ code. This will automatically scan for installed plugins when it is constructed, so then we just use a for loop to get the details of each plugin, creating a ScannedPlugin object for each and adding it to the plugins list.

You'll need to replace USERNAME with your username on your system

plugins = []

# Scan the plugins
scanner = PluginScanner()

print(f"Scanned {scanner.getNumPlugins()} plugins")

# Create an array of ScannedPlugin so we can write it to the cache file
for pluginIndex in range(scanner.getNumPlugins()):
    plugin = ScannedPlugin(scanner.getName(pluginIndex),
                            scanner.getManufacturer(pluginIndex),
                            scanner.getFileOrIdentifier(pluginIndex))

    plugins.append(plugin)

# Go through each plugin, check if it appear in the tagset
databasePath = "/Users/<USERNAME>/Music/Audio Music Apps/Databases/Tags"
warningCount = 0


Next we need to open a log file and loop through each plugin object we've created. For each plugin we do the following: Once we've looped through all the plugins we post a notification, and then call suppressStream to suppress the unnecessary output JUCE code produces on exit. Let's at this code below everything we've got so far:

with open(os.path.join(ROOT_DIR, "log.txt"), "w") as logFile:

    for plugin in plugins:

        # Work out the tagset file name for this plugin
        tagsetFileName = buildTagSetFileName(plugin.getIdentifier())
        tagsetFilePath = os.path.join(databasePath, tagsetFileName)

        logPrefix = f"\n{plugin.getManufacturer()} - {plugin.getName()} ({tagsetFileName}) : "

        try:
            # Check the plugin has a tagset
            if os.path.isfile(tagsetFilePath):

                # Get the categories in the tagset
                categories = getCategoriesFromTagset(tagsetFilePath)

                if len(categories) == 0:
                    logFile.write(logPrefix + "has no categories")
                    warningCount += 1
                elif len(categories) > 1:
                    logFile.write(logPrefix + f"is in: {categories}")
                    warningCount += 1
            else:
                logFile.write(logPrefix + "has no tagset")
                warningCount += 1

        except TagsetError as e:
            logFile.write(logPrefix + f"failed to parse tagset ({e})")
            warningCount += 1

        except Exception as e:
            logFile.write(logPrefix + f"failed to process ({e})")
            warningCount += 1

postNotification(title="AU Checker",
                content=f"{warningCount} warnings, {len(plugins)} plugins scanned")

# Redirect stdout and stderr otherwise JUCE will produce tons of output on exit
suppressStream(sys.stdout)
suppressStream(sys.stderr)


Now our script is done, running the command python main.py will take a few minutes and then populate log.txt with the results and post the notification.

Scheduling with launchctl

MacOS provides a way to schedule scripts using a tool called launchctl, so to make our script run each day we just need to make launchctl aware of it.

launchctl can either call the python script directly, or if you use an anaconda or virtualenv environment for your Python installation, launchctl can call a bash script which then sets up the environment and calls the Python script.

If you use a virtual environment you should now create a script which we'll call launchd.sh. Run chmod +x launchd.sh to make it executable. Below is an example of the code required to make this work for an anaconda environment:

#! /bin/bash

PATH=${PATH}:/Users/<USERNAME>/anaconda3/bin

source activate mlEnv_py36

python /absolute/path/to/main.py


Now for the final step, in the directory ~/Library/LaunchAgents create a file called com.<USERNAME>.auchecker.plist and copy the below XML into it. The script command (all caps below) should either be "python path/to/main.py" or the path to launchd.sh if you are using the bash script.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>com.<USERNAME>.auchecker</string>
        <key>Program</key>
        <string><SCRIPT COMMAND GOES HERE></string>
        <key>StartCalendarInterval</key>
        <dict>
            <key>Minute</key>
            <integer>00</integer>
            <key>Hour</key>
            <integer>20</integer>
        </dict>
    </dict>
</plist>


You can now set the minute and hour integer values to the time you would like the script to run, and load the configuration into launchctl using the following command:

launchctl load -w ~/Library/LaunchAgents/com.<USERNAME>.auchecker.plist


And that's everything! You can check that launchd has loaded the configuration by running the following command and checking that the script runs correctly.

launchctl start com.<USERNAME>.auchecker


Conclusion

So now you should have a completed Python script that can scan for plugins, interpret tagset data created by Logic Pro X's plugin manager, and generate notifications based on that data.

This is really just a start and there is much more you could do, such as programatically editing the tagset data after interpreting it or creating an interactive tool to organise the data.

Hopefully this series of posts has been useful and provided the basics of how to get started with the topics that have been covered.