ChatGPT - ML Augmented Development (2 of 3)

In the previous blog we talked about how ML Augmented Development can be used to accelerate the development process. Now we’ll talk about ChatGPT. You’ve probably heard about this tool on a news feed, or maybe you’ve even tried it out. This is one of the most exciting things here. In this blog I’ll review what ChatGPT is, tests that I’ve run, and what are the pros and cons of ChatGPT as a tool for programming.

What is ChatGP

From the openai.com Site

We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.

https://openai.com/blog/chatgpt/

 

At its simplest, ChatGPT is the smartest Chatbot you will ever work with. It can answer questions, write papers, explain things. I have seen ChatGPT used to write papers, to write scripts for Ads, to answer questions about programs. 

From the matrix in the first blog. ChatGPT is a Prompt:Holistic ML Tool. 

Experiments: 

To experiment with these ML Tools, I experimented in a few ways: 

• I asked them to create a well-known experiment: create a random walk
• Replicate a function we were working on at Metal Toad: get a message out of AWS SQS and send it to MailChimp. 
• Create unit tests for the MailChimp Function. 

 

Experiment 1

Prompt: Write a program in Python to simulate a random walk

Response: 

Here is a simple Python program that simulates a random walk:

import random


# Number of steps in the random walk

num_steps = 1000


# Starting position

position = 0


# Generate the random walk

for i in range(num_steps):

 step = random.choice([-1, 1])

 position += step


print("Final position:", position)

 

This program uses the random.choice() function to randomly choose between taking a step to the left (-1) or to the right (1) at each step of the walk. The for loop iterates for a specified number of steps (in this case, 1000), and the final position is printed at the end.

 

You can also visualize the random walk by plotting the position at each step using a library such as matplotlib:

mport matplotlib.pyplot as plt


# Number of steps in the random walk

num_steps = 1000


# Starting position

position = 0


positions = [position]


# Generate the random walk

for i in range(num_steps):

 step = random.choice([-1, 1])

 position += step

 positions.append(position)


plt.plot(positions)

plt.show()

This will show a graph of the position of the walker over time.

Observation: It’s worth noting at this point that this was the only example I did with ChatGPT that worked with zero changes.

 

Experiment 2:

Prompt: Using Python write a lambda function to send a message to pull a message from SQS and sent it to MailChimp.

Response:

Here is an example of a Python lambda function that pulls a message from an SQS queue and sends it to MailChimp using the boto3 library for AWS and mailchimp3 library for MailChimp:

import boto3

import json

from mailchimp3 import MailChimp


sqs = boto3.client('sqs')

queue_url = 'YOUR_QUEUE_URL'


mailchimp = MailChimp(mc_api='YOUR_MAILCHIMP_API_KEY', mc_user='YOUR_MAILCHIMP_USERNAME')

list_id = 'YOUR_MAILCHIMP_LIST_ID'


def lambda_handler(event, context):

 # Receive message from SQS

 response = sqs.receive_message(QueueUrl=queue_url)

 if 'Messages' in response:

 message = response['Messages'][0]

 body = json.loads(message['Body'])

 email = body['email']

 #Add email to mailchimp

 mailchimp.lists.members.create(list_id, {

 'email_address': email,

 'status': 'subscribed'

 })

 # Delete the message from the SQS queue

 sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message['ReceiptHandle'])

 return 'Success'

Observation: The explanation is very complete. I didn’t want to worry about security issues so I asked ChatGPT to handle that for me. 

 

Experiment 3:

Prompt: Modify the above to pull the variables YOUR_QUEUE_URL, YOUR_MAILCHIMP_API_KEY, YOUR_MAILCHIMP_USERNAME, and YOUR_MAILCHIMP_LIST_ID from AWS Secret Manager

Response: Here is an example of how you can modify the above lambda function to pull the variables YOUR_QUEUE_URL, YOUR_MAILCHIMP_API_KEY, YOUR_MAILCHIMP_USERNAME, and YOUR_MAILCHIMP_LIST_ID from AWS Secrets Manager:

import boto3

import json

from mailchimp3 import MailChimp


# Create a session to interact with Secrets Manager

session = boto3.session.Session()

client = session.client('secretsmanager')


# Get secrets from Secrets Manager

def get_secret(secret_name):

 response = client.get_secret_value(SecretId=secret_name)

 return json.loads(response['SecretString'])


secrets = get_secret('YOUR_SECRET_NAME')


sqs = boto3.client('sqs')

queue_url = secrets['queue_url']


mailchimp = MailChimp(mc_api=secrets['api_key'], mc_user=secrets['username'])

list_id = secrets['list_id']


def lambda_handler(event, context):

 # Receive message from SQS

 response = sqs.receive_message(QueueUrl=queue_url)

 if 'Messages' in response:

 message = response['Messages'][0]

 body = json.loads(message['Body'])

 email = body['email']

 #Add email to mailchimp

 mailchimp.lists.members.create(list_id, {

 'email_address': email,

 'status': 'subscribed'

 })

 # Delete the message from the SQS queue

 sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message['ReceiptHandle'])

   return 'Success'

You will need to replace YOUR_SECRET_NAME with the appropriate value for your secret in AWS Secrets Manager.

In this example, the function uses the boto3 library to interact with AWS Secrets Manager, and the get_secret() function is used to retrieve the secret values for the specified secret name. The secret should be in json format, with keys matching the variable names used in the code.

The function then uses these variables to configure the SQS and MailChimp clients as before. This way, you can keep the sensitive information(secrets) away from the code and easily rotate or manage them using AWS Secrets Manager.

 

Observation: This is just an example, you might want to add some error handling, validations and other customizations based on your use case.

This didn’t work the first time I attempted to run it. While it compiles and runs, it doesn’t work because our SQS has different fields and our function is for notifying of unread messages. So we need to insert a check that the SQS message is still unread in the database. 

Tha said this took 15 minutes to write, and the structure was good. I may have been able to phrase a prompt to ChatGPT to get these added checks and tasks. Getting the phrasing right would take more time than writing the code or maybe I ask ChatGPT to create a separate function to do that task and add it. 

 

Experiment 4:

Because of how ChatGPT works, I also tried having it troubleshoot a program:

Prompt: what does this error mean:

test/test_lambda_handler.py:15: in <module>

 from app import lambda_handler

app.py:7: in <module>

 mailchimp = MailChimp(mc_api=secrets['api_key'], mc_user=secrets['username'])

E NameError: name 'secrets' is not defined

Response:

his error message is indicating that there is an issue with the lambda_handler function in the app.py file. Specifically, the error is occurring on line 7, where the mailchimp variable is being defined. The error message states that there is a NameError, meaning that the variable secrets is not defined.

It seems that the secrets variable, which is used to store the mailchimp credentials, is not being initialized or imported correctly. It might be missing or not imported correctly.

This error means that the code is trying to access a variable named secrets, but it is not defined in the current scope. It could also mean that the secrets variable was defined in a different file or module and is not being imported correctly in the current file.

It could also mean that the secrets were not correctly retrieved from the AWS Secrets Manager, and the function is trying to access the secrets that are not yet initialized.

You should check the import statements and the retrieval of the secrets, to make sure the secrets are imported and initialized correctly, and that the application can access them.

Observation: The ability to parse error messages, even an obvious one, and explain it helps developers quickly fix problems.

 

Findings:

Bad - ChatGPT is not good at small modifications to files unless I stated them as a simple statement. It can create code that is wrong. To varying degrees, and needs to be reviewed and tested. 

Good - ChatGPT is good at building large blocks of code where the functionality can clearly be defined, because of its conversational nature and the depth of its knowledge, it can provide a pair programming experience when another developer isn’t available to pair.